## Taming an Inclinometer

A project we are working on requires a MEMs inclinometer to assure coplanarity between a flat surface and a sensor.

The Problem

The sensor has a high gain section before an analog-to-digital converter feeds data to the microprocessor. Unless the sensor is located in a Faraday cage, the 120 hz hum generated causes severe instability in the digital data. It was determined that the center-line of this signal is the stable signal output.

At the extreme high and low outputs, the AC signal is compressed, but the center-line signal is still correct.

The code takes 20 asynchronous X and Y axis readings, finds the highest and and lowest, determines their difference, and divides by two to get the current X and Y inclination.

This code snippet just covers the search for highest and lowest value. We have left off the hardware code as it is different for each platform.

``````    tiltDataXhiAccum = tiltDataX[0];
tiltDataYhiAccum = tiltDataY[0];
tiltDataXloAccum = tiltDataX[0];
tiltDataYloAccum = tiltDataY[0];

//this one finds the highest value
for ( j = 1; j < 20; j++ )
{
if ( tiltDataXhiAccum < tiltDataX[j] )
{
tiltDataXhiAccum = tiltDataX[j];
}
if ( tiltDataYhiAccum < tiltDataY[j] )
{
tiltDataYhiAccum = tiltDataY[j];
}
}

//this one finds the lowest value
for ( j = 1; j < 20; j++ )
{
if ( tiltDataXloAccum > tiltDataX[j] )
{
tiltDataXloAccum = tiltDataX[j];
}
if ( tiltDataYloAccum > tiltDataY[j] )
{
tiltDataYloAccum = tiltDataY[j];
}
}

tiltDataXAccum = ( tiltDataXhiAccum - tiltDataXloAccum ) / 2;
tiltDataYAccum = ( tiltDataYhiAccum - tiltDataYloAccum ) / 2;``````

This solution allowed us to go from a realtime bobble rate of over 5 per second with differences of 10 or more, to a stable reading that might change every few seconds with a worst case difference of 1.

## Designed to Wear Out / Designed to Last – Pt. 3

Here we are going to take a look at current technologies’ strengths and weaknesses in relation to the past.

## Increasing Active Elements

For perspective, there has been a monumental increase in the number of active elements available in an integrated circuit compared to decades ago. This has allowed an increase in device complexity and speed.

Because of this complexity, there has been a necessary increase in the complexity of testing and test tools. Modern chips have many redundant operating sections and a JTAG ( test bus ). This allows all sections on a chip to be 100% functional tested. Sections found to be bad are mapped out. This has become necessary because, at current densities, not all of the elements on a newly manufactured IC are functional. Failed functional units are mapped out using methods such as read-only-memory elements or other means. Without this practice, the cost of a working chip would be too high. Above a certain threshold of failed sections, the chip is simply discarded. This practice started as densities rose in memory chips. There were always failed banks of memory elements, so building redundant elements and mapping them in or out made sense. Otherwise the chip yield would be too low. Chips with mapped out sections could be sold as lower density chips, assuring some revenue, especially at the time a product was new and yields were low.

## Increasing Pin Count

An increase in complexity also means an increase in die size and the number of pins to a single IC package. When I started in this business, 40 pins was the largest number. The largest number these days is the 3647 FCLGA package. That indeed is 3,647 pins !

## Decreasing Repairability

Inevitably, this pin count increase means that replacement of a single XSI (extreme scale integration) chip is nearly impossible, and certainly not feasible. As far as chip lifetime or reliability, this is unknown, other than the MTBF claimed in the datasheet. Electromigration is the likely cause of long term chip failure in these instances. The circuit board on which you find high count ICs is a throwaway item.

## The Pause in Moores’ Law

No doubt, something will come along to allow Moores’ law to again provide speed and density improvements, but for the last 5 years (as of April, 2020), we have been stuck at a particular minimum feature size of a transistor on a modern processor chip. There is a good deal of work on making more efficient designs of logic elements, but currently nothing close to the multiplicative improvements of Moores’ law.

I mention Moores’ Law in the faint hope that cooler heads will prevail upon us to, at least, make complex ICs reusable/reprogrammable.

Note: A short list of possible improvements

1. 3D Stacking of elements
2. New materials
3. Innovative logic configurations

## Designed to Last

There are many examples of technologies which are designed to function for decades into multiple centuries.

What goes into an item that can make it last decades or even centuries ? Here at LCM, a lot of the original logic we are running in our collection, was not rated as long life when originally used in our machines. Expected lifetime data had the best of them lasting for a few decades, maximum.

This has generated a set of rules we abide by: If it looks like stone or amber, it can last a long time. If it looks like plastic (or is plastic), it is not going to last. One particular exception is epoxy packages. Certain epoxy resins are similar in structure to tree resins, which so far, hold the record ( millions of years ) for preserving ancient insects, pollen, and plants. )

1. Semiconductors and Integrated Circuits – Life data in databooks for IC expected lifetimes from the era in which they were created, typically have these devices failing more than a decade ago. Yet here, they are still performing their function. Staying functional tends to favor ceramic packages (stone) and specific epoxy packages (amber)(See “Notes on Epoxy Packages for Semiconductors” below.
2. How Much Heat – Systems designed to have a higher internal ambient operating temperature, tended to have a much higher failure rate. Datasheets at the time ( and today ) show a direct correlation between ambient operating temperature and operating life.
3. Cooling Fans – These sit right in the middle of the life curve. Although the bearing has a low wear rate, 30 years seem to be the upper limit. ( It would be cool if someone could come up with a frictionless magnetic bearing.)
4. A Special Note About Fans and Heat: A lot of our power supplies have an intimate relationship with fans, meaning, if the fan fails, the power supply will fail, as well. When we re-engineer a power supply to replace an older or failed unit, we specify that the new supply can keep operating even though the fan has failed. This is accomplished by the fact that the replacement power supply components have a higher efficiency ( thus generating less heat performing their function ) and can tolerate heat better than the old power supply components. ( This has the added advantage of lower air conditioning costs )

## Winners In The Longevity Game

The absolute winners we have found and utilized in our systems are what are known as “bricks”. These are power supply modules which are fully integrated. They come in various sizes which determine their power ratings. Full bricks top out around a kilowatt. Half-Bricks are around 500 watts. Quarter-Bricks around 250 watts.

Lifetimes (MTBF) at full load and temperature for “bricks” go from around 40 years to around an astounding 500 years. ( That is not a typo. The part in question is a Murata UHE-5/5000-Q12-C. The whole UHE series has this rating. Price \$61.90) These devices, as you may have already guessed, are epoxy encapsulated.

## Designed-In Longevity

This refers to what we have encountered upon restoring the machines in our collection. There is a definite intent at work when one examines the component choices. For example, DECs PDP-10 KL series power supplies have filter capacitors at four times the necessary capacitance. The amount of capacitance declines with age pretty linearly till the end of lifetime (around 14 years). This means these particular components will still allow power supply function 4 times longer. That’s at least 3 times beyond the machine’s commercial life rating ( 5 -7 years). We got these machines 15 to 20 years after their last turn-on, and they ran for most of a year before we had cascading failures of the filter capacitors.

## Notes on Epoxy Packages for Semiconductors

Epoxy packaging for semiconductors became popular in the mid-1960’s. It replaced ceramic, as it was less expensive. Epoxy, unfortunately, can be made with different resins and other ingredients that give it different material characteristics. There is a correlation between cost and moisture intrusion. Lower cost, more moisture. This gave epoxy a bad name as it was used to make ICs’ more competitive in the market. This led to a number of market loss moments for certain manufacturers, as the moisture intrusion occurred at a predictable rate depending on ambient humidity for a particular region of the country.

It was a multi-faceted problem. The moisture intrusion occurred where the IC lead connects with the package. Moisture intrusion into the epoxy and poor metal quality ( tin alloy ) of the lead frame causes corrosion of the lead, which allows moisture into the IC cavity and changes the bulk resistivity of the IC die. The electrical specs go off a cliff and the IC fails.

( Note: If the lead frame had been made from a different alloy or the epoxy was a higher grade, this failure had little or no chance of occurring. I find it hard to justify the cost differential given the ultimate cost to the end users and the manufacturer )

(I was a field engineer in the mid to late 1970’s and spent many an hour replacing ICs with this problem.)

The only epoxy packages that have made it to the present day used better materials and thus are still functional. There are thousands functioning ICs on circuit boards in the Living Computers collection heading toward their 50th anniversary and a spares inventory of thousands.

## Mechanical Switches

Whether toggle, pushbutton, slide, micro, or rotary, mechanical switches are all over the lifetime map. In the commercial world, switches are rated at the maximum number of actuations at a specified current. For the most part, the switches I have encountered meet or exceed the actuation specification. There is a limitation, though.

Aging of Beryllium Copper

If the internal mechanical design use a beryllium copper flat or coil spring, it has almost surely failed by the time we at the museum have encountered this type of switch. Beryllium copper goes from supple and springy to brittle after 30 years or so.

The result, as you may have guessed, is a non-functioning machine due to switches that operate intermittently or not at all. ( We had a whole line of memory cabinets, with hundreds of bright shiny toggle switches for memory mapping, that wouldn’t function till we replaced the switches.

Circuit Breakers

These fall into the beryllium copper spring family, so they are pretty much failed, or in addition to not closing, their trip point has typically shifted so you get a premature trip or a trip well above the trip point ( sometimes no trip and the protected circuit burns up ).

Slide Switches

We’ve had one surprising winner in the switch longevity department, and that is slide switches. With few exceptions ( usually due to mechanical damage to the sliding element ), an intact slide switch can be quickly resurrected with cleaning and a little light oil.

Rotary Switches

The runner up in the longevity game is the rotary switch. Typically all a wonky rotary switch needs is a spray from an alcohol cleaner and contact lubricant, and it functions, no matter what the age. ( we have hardware going back to the 1920’s whose rotary switches are still functional ) You can consider the switch failed if the contact wafer is cracked or broken. My guess as to longevity involves the phenolic wafer getting significant mechanical strength by being riveted.

Relays

These devices are essentially an electrically actuated switch. Their most common configuration is some kind of leaf spring. If the spring is beryllium copper based, you of course, have a failed relay 30 years hence. Rotary relays tend to be fairly reliable, but require a little more maintenance to keep them going. Mercury wetted relays are fairly reliable ( we still have some running in a couple of pieces of hardware ), but are not recommended because of their mercury content along with the mercury being contained in a fragile glass envelope.

## Designed to Wear Out / Designed to Last – Intro

This is an article series whose purpose is to shine a comprehensive light on one important aspect of technology that only gets passing mention: Our ability to determine how long a component or system can function based on the engineering decisions made in the interests of monetary consideration and/or reliability. This is especially germane today, as decisions about discarding a technology item at “end of life” now impinge on how much toxic waste we are loading the environment with. ( I have yet to find an “obsolete” cellphone or computer that didn’t work perfectly when discarded, for other than mechanical or liquid immersion damage. ) The definition of obsolete depends on who you ask. The new smartphone you buy today is only a few percent actual new technology compared to the old smartphone your are discarding.

So we seem to have uncovered the operating model that most companies producing and selling technology have adopted:

The model for tech from approximately the 1920’s to today involves having key components in your product which have a known MTBF ( mean time before failure ). Thus you can predict a known replacement rate for your product. Start a second product stream offering replacement parts for the ones you know are going to predictably fail.

Unless or until the majority of your users are technically savvy enough to realize this is a way to keep selling products in a saturated market, one is assured of continuing and predictable sales and profit

## Vacuum Tubes

One could argue that vacuum tubes were inherently unreliable, so they made them as robust as possible and just accepted their limitations. This turns out to be a bit wide of the truth.

In a step-wise fashion, the manufacturing process evolution for the vacuum tube went something like this:

1. The first vacuum tubes were handmade with limited production.
2. As soon as production ramped up to feed demand, it was inevitable that the manufacturer would tweek the processes used to make a vacuum tube to minimize the amount of time it takes to assemble the finished product from raw materials, to ready to assemble, and assemble (using hands, jigs, and other production equipment).
3. As soon as all of relevant costs have been driven out of the system of producing vacuum tubes, the manufacturer rightly looks to other means to make a profit. (Process tweaks and labor input reductions) Sooner or later the ecology of manufacturers who produce vacuum tubes reaches an equilibrium which, despite their best efforts, doesn’t allow any greater profitability.
4. Product lifetime now looms large. Sell more product over time, make more profit.

( Note: this describes the manufacturing process evolution for most “tech” products. )

## Ways To Sell More Vacuum Tubes

1. I only learned about this particular lifetime determinant recently. It seems that in order to get the filament temperature high enough for electron emission, the tungsten had to be alloyed with thorium, which raises the melting point to that of a tungsten/thorium alloy (well above the temperature where the alloy emits electrons. The amount of alloying is directly proportional to the time a DC current-over-time applied in a plating tank. The tube life is directly proportional to the time it takes for the thorium to boil off. More thorium – More lifetime. As the final boil-off occurs, the filament temperature is rises above the melting point of tungsten which results in the filament overheating and burning out. Vacuum tubes manufactured for the US Military were larded up with thorium, and thus met the extended lifetime specifications the US military demanded.
2. getter is a deposit of reactive material that is placed inside a vacuum system, for the purpose of completing and maintaining the vacuum. ( https://en.wikipedia.org/wiki/Getter ) The quality and amounts of various elements is adjustable by the manufacturer and will determine vacuum tube life. I addition, you can get a short lifetime tube just by eliminating the getter. (We have plenty of examples without getters)
3. In order assure that parts were available as they wore out “normally”, the tube manufacturers put a tube tester (stocked with replacement tubes ) in every convenient location throughout a geographic area. (Drug and hardware stores typically)
4. Make and sell more end products using the same (or slightly improved) technology. This is accomplished by the “Longer, Lower, Wider” paradigm used by the auto industry (and now applied to radios, televisions, etc)starting around the 1940’s. This involves making a greater variety of the same (or slightly modified) product in a different package year to year. Anyone who has read those old ads can see this methodology in action.

Some of the environmental results of the vacuum tube era were:

1. Mounds of glass along with refined metals (tungsten, thorium, steel, and others) and minerals (mica) added to landfills. It is unclear if any of the glass was recycled.
2. Some of the metals and minerals were leached into the ground by rain.
3. Large amounts of scrap-metal ( chassis ) and wood ( cabinets ) ended up in landfill and were either recycled or rotted into the ground.

## Mainframe Computer Connector Failure and Solution

We have uncovered another interesting phenomena which has impacted our collection.  As the subject line indicates, we have identified a failure mode concerning the gold finger contacts on the numerous circuit boards in our large systems.  Specifically, a hard, mostly carbon (with an amalgam of other unknown atmospheric particles) film has adhered selectively to the plated gold.  It is highly resistant to cleaning with anything from alcohol to acetone.  The only way to remove the film is with mild abrasion ( 3M Scotchbrite ).

When the collection was first started, the large mainframes suffered from reliability issues that was subsequently tied to buildup of soot and dust on the gold finger contacts of the printed circuit boards. This was back in 2002, and alcohol cleaning did not entirely solve the reliability issue as the intermittency recurred after several months. In doing our research, we found a product online called “Stabilant 22”.  It is a polymer substance that becomes conductive when a voltage differential occurs between two planes of metal, as between the contact fingers of a printed circuit board and the PCB connector. When we applied it, the problem was solved and did not recur for several years.

In the last couple of years, two of our large mainframes ( and other machines ) have became unstable once again. But this time, alcohol cleaning and application of Stabilant 22 has not fixed the problem. While evaluating the nature of the problem, I experimented by applying mild abrasion to the circuit board contact fingers, and then cleaned them with Kim-Wipes and alcohol. Previously, where no visible residue was left on the wipe, there was now a significant quantity of dark residue removed.

Once the contacts were again treated with the contact enhancer, the intermittent failures went away.

We applied this fix to all of the circuit cards in both machines and re-established the robust reliability we had experienced previously.

Our speculation for this phenomenon is that the carbon deposited on the PCB contact fingers is in the form of 2D graphene glass, which is a good insulator and quite transparent. The abrasion breaks the surface of the glass and the carbon converts to the particulate form we are used to which is easily removed with alcohol.

The step by step process we have developed to successfully restore the contact integrity is:

1. Use an alcohol soaked KimWipe to first clean the contact of loose surface material.
2. Lightly rub the contact fingers with a super fine grade Scotchbrite pad.
3. Use an alcohol soaked KimWipe to remove the carbon rich material shown in the photo.
4. Repeat step 3 till the wipe no longer accumulates material.

So far, we have reached out to a number of resources for a definitive determination of the nature of the material we are cleaning off, with no confirmation. We invite the assistance of the technical community at large to help us with this determination.

## IMLAC PDS-1 Power Supply

In early July of 2019, the power supply of one of our rarest and iconic machines, started to fail. This is the IMLAC PDS-1 originally produced from 1970 to 1972. Despite the efforts of our staff to troubleshoot and replace components, we were soon left with a completely failed power supply.

Typical of these situations, we set about to do an engineering evaluation toward designing a form, fit, and functional replacement. The photos below show the power supply system in its’ original form.

Accessing the schematics, we found what voltages and currents the power supply system had to provide. Next, using the power supply components we have available for this purpose, we had to design a system which fits in the chassis and interfaces with the control and power signals the computer needs to run.

Surprise ! The Power Supply Generates a Non-DC timing Signal

The schematic below shows a section which we couldn’t figure out initially. At first glance, the collection of four diodes on the left looks like a bridge rectifier. On closer examination, the anodes and cathodes are not hooked up like a bridge rectifier. What we have here instead is a frequency doubler which is used to generates a 120 hz signal from the 60 hz power line that is used as a periodic interrupt for the video display logic. Not at all expected.

Below is our replication of this circuit. We used a miniature 120 VAC to 10 VAC transformer.

After the surprise above, we set about removing the original power supply components and installing a new configurable supply along with the necessary modifications to the internal wiring harness.

Below are two images showing the right and left power supply chassis as modified.

We completed and tested the above modifications in about 1 1/2 weeks. A week after the unit was put back in service, a third power supply ( located in the control console ) also failed. We replaced it with another configurable supply as shown below.

The system has now been running since late July without incident.

This is a good example of the type of work we have been performing for the last 15 years.

## Bendix G-15 – Solder Degradation

In the process of restoring the Bendix G-15, we have discovered a phenomena that degrades the electrical connections which provide bias and signal flow, rendering the computer non-functional.

## Failed Connections

Below is a group of photos which illuminate this failure mode called “electromigration”. This process is caused by a continuous DC potential applied to a metal junction. Metal ions migrate in the direction of current flow. For most new machines, this is not a problem as this process takes quite a number of years to progress to the point where the electrical connection is broken. At LCM+L, we get machines after they had run for a long time. Worse yet, since it is our intent to restore and run the machines for as long as we can, it is necessary to find a solution that allows that maintenance need only be done every decade or so.

## Failing Solder

This same phenomena plays out in the metal structure of the solder itself. The photos below show the before and after of solder restoration. In the first photo, the solder looks dull and mottled. This is due to the tin having migrated out leaving only lead in the Tin/Lead solder formulations used until the early 2000’s. The modern formulations are Tin/Silver/Copper and are much less likely to have metal ion migration.

## Long, Repetitive Work

This restoration process took quite a while. After determining that all the tube modules in the machine were affected in this way, we simply set about removing a module and then removing and replacing the solder in all the high, continuous current sections.

An interesting article on solder, covering some of the topics mentioned in the article can be found at: https://en.wikipedia.org/wiki/Solder

## Xerox ALTO – Interesting Issue

In the process of restoring the Xerox ALTO, an interesting issue came up.

Background

We received our first ALTO in running condition and after evaluation and testing, put in on the exhibit floor available to the public. One afternoon about a year later, the machine suddenly froze and stopped functioning. It was taken off the floor and evaluated in one of our labs. When it became clear that power supply current was not flowing into random parts of the backplane, the focus shifted to the power supply rails. It was there we were confronted with this phenomena.

The ALTO Was A Prototype

Certain production and test details were left out of the ALTO. The amount of current running through individual pins supplying regulated DC to the logic is unusually high. Most of the time, power supply current is fed through as many pins as possible to reduce the total current running through any one pin. Because the ALTO was a prototype, the designers only used the minimum number of pins to do the job. This resulted in a phenomenon called “electro-migration”. It is the reverse of the process used for electroplating. In this instance, tin ions migrate away from the solder joints carrying the power supply current. The six pins in the center of the first photo show a mottled (instead of smooth) surface, and one of the pins has a dark ring around it indicating where the solder has totally migrated away from the connection (lower right pin). The second photo show six pins where the tin has migrated away.

Confronted with the preventing this in the future, LCM Engineering increased the surface area of the connections by soldering brass buss rails to all of the ALTO backplane power supply pins. This is shown in the photo below:

Once this fix was applied, the ALTO was put back in service, and this phenomenon has not repeated itself.

The ALTO will be monitored to see if this phenomenon shows up again. This chapter has also been instructive for some of the other machines we are restoring. In these instances, it is extreme age, rather than something done for a prototype as the causative factor.

## Bendix G-15 Vacuum Tubes

Early in the restoration and troubleshooting of the Bendix G-15 it was noted that tube filament failures occur with some regularity. It is not possible to observe working filaments on all the tube modules, as at least half the tubes have what is call a “getter coating” at the top of the tube, obscuring the filaments.
We hosted a subject matter expert to aid with troubleshooting the G-15, and he indicated that tube filament failures were the principal cause of machine downtime, usually about once per week. This invariably entailed up to a day of troubleshooting to find the offending tube(s).
Due to the above information and our own experience, it was decided to engineer a sensor and indicator system which would allow quick identification of the offending tube or tubes.
The configuration decided upon was a hall effect sensor coupled to a passive magnetic field concentrator ( wound ferrite core ) placed in the current path of each individual vacuum tube filament that would light an led when the tube filament was functional. Up to six sensors ( the largest complement of filaments in a tube module ) are packaged on a substrate which fits on each tube module and are powered by the filament voltage entering each module.

We gave the sensor package the acronym “FICUS”. It breaks down to FI = Filament, CU = Current, and S = Sensor.

Here is what a hall effect sensor mated to a wound ferrite core looks like:

Below is a photo of a FICUS module in the process of being assembled:

Here is a completed FICUS module:

Here is the FICUS after the wires have been attached to the vacuum tube module:

Here is the completed vacuum tube/FICUS module ready to be plugged into the Bendix G-15:

Here is the view of the Vacuum Tube/FICUS module oriented as it would be in the G-15:

And finally, a couple of Vacuum Tube/FICUS modules in an operating G-15:

## DEC Computer Power Supply Module Retrofit

In the process of troubleshooting our earliest machines, we had to replace large components called electrolytic capacitors. These are located in all the power supplies for any computer. We successfully replaced these devices and got the machines running. Recently though, we have started to see these devices fail once more. They have a finite life of a maximum of 14 years. That means that we have to replace these devices every 10 to 14 years. Also, the larger capacitors are no longer manufactured, but can still be special ordered. As it is our mission to have our computing hardware last for a lot longer than that, we did our research and engineered a replacement for the power supply modules these capacitors are found in. Our goal was to provide several decades of service without having to service these modules. The photos and descriptions below show the process:

Below is what the original power supply circuit board looked like

When we strip out the circuit board and remove the heat sink, we get this

We created, using a CAD program and a 3D printer, a plastic component mounting for the new components.

As you can see, the plastic mount fit perfectly into the old power module frame.

After populating the mount with all the components it looks like this.

Now we attach the modified heat sink to the original module frame.

Install the assembled component mount in the frame along with the modified heatsink and the new power module is complete.

One of the features of the module is, it has no solder connections, all of them being compression.  Wires are compressed into a square cross section using a stainless steel screw.  This provides very high reliability.

The upshot of all this work ( there were 38 modules in various machines ), is power supplies that are more efficient and have a rated MTBF ( mean time before failure ) of 40 years. These power supply modules draw 2/3 less power and produce 2/3 less heat, reducing the heat load on all the components in a machine. In addition, as a result of these changes, the total power savings per year is 250,000 kilowatt hours. Electricity rates in this area of Seattle are about 8 cents/kilowatt hour. That means a direct cost savings on our electric bill of \$20,000 a year.