It was early 2017 when Google implemented automatic programming, the concept of programs creating other programs to perform a specific task, into their Artificial Intelligence (AI). Automated algorithm design (AutoML or automatic machine learning) could generate code with performance numbers drastically higher than their man-made counterparts. Google applied the code to image identification and the accuracy of detection reached record values. This paved the way for the rise of TensorFlow and tensor processing units, ASICs specifically tailored for machine learning applications.

The tech is still fresh from the oven and there is much room for change and further development. Google's A.I. - the progenitor of the superior optimized neural network, could have added an additional imprint with instructions for reiterative self-replication. This does not benefit the machine learning application, as it would simply saturate memory without another part of code to counteract the loop. But it plays a crucial role in a self-sustaining A.I. - one that can think, act, and develop on its own (the entire process of creating an A.I. is executed and refined without human intervention).

The prerequisites are all of the above - with stimuli from pertinent sources unique to the application. For the image processing example, stimuli can come from multiple real-time cameras.  In memory, there may be 20 co-existing copies of A.I., each programmed to erase itself after a few trillion clock cycles (is this long enough?) to avoid overflow. When a new object is perceived from the cameras (say, in a jungle), the A.I. can adjust its metrics for that object by training itself given the video frames from the real-time cameras. It can assign a new code-name to that object (later replaced by a weirder scientific name by some biologist or whatever).

Moving Ahead: the FPGA counterpart for a TPU

Given these presumptions and premonitions, it is beneficial to the self-sustaining A.I. to possess bounded flexibility in adjusting it's hardware/monolithic design. This is done by providing a source of stimulus from the ASIC design itself. It is difficult to make assumptions with limited knowledge on tensor processing units, but there must be an incentive when an ASIC is designed for this kind of A.I.
This implies that the tensor processing unit must not be rigid (i.e. it is similar to having the FPGA equivalent in A.I.) 

A possible ramification from this technology is an adaptive instruction set architecture (ISA) for the TPU.

As a simple example, with the set of instructions above, a self-sustaining A.I. may want to add an instruction as a means of developing itself. Maybe it would need to make adjustments to its matrix multiplier unit, and do a more advanced convolution operation. A consequence of this is that such changes may not be tracked by the human developer (because it is self-sustaining) and when a problem occurs, debugging may prove to be an impossibility.

Keeping an original copy of an A.I. is a solution. There could be another A.I. trained for troubleshooting such scenarios, if they were ever to occur. This time there are 2 inputs, the original A.I., and the A.I. that has evolved/changed over time. The troubleshooting A.I. will then discern a unique problem as it passes through its neural network. Thus, another vulnerability is found, where an unlisted problem (not in the training set) and with low correlation values to other problems could render the original problem intractable.

There are many abstract ways to deal with this kind of future technology. Who knows? Maybe  self-sustaining A.I. could build code that can solve the problem itself.