HVP Automation NISSorter

Nut in Shell (NIS) Sorter/Scanner

April 2018 - Initial Accuracy Testing

May 2017 - System is UP! video

Description

I have been working on this for some time and it's finally up! While vision sorters for nuts have been available for many years they are mostly fairly big and targeted at high volume moderate accuracy applications. This sorter is compact and designed to be low volume high accuracy. To get to commercial volumes it is necessary to link several together and/or arrange the installation so they can run 24/7.

In addition to acting as a sorter it can be used as a lab scanner to assess the quality of samples and results from multiple machines can be compared directly. This might be used in research situations or commercially it could be used to characterize volume sorters and then given a sample of a new batch the optimum settings for the volume sorter could be determined before the actual processing for the batch began.

Availability

The sorter can probably be made available at any time provided there is enough interest. It would probably need four or five Australian advance orders to make producing them worthwhile. Email me if you are interested.

Method of analysis and decision making

In the early days of vision sorters they worked by simply inspecting each pixel in the data stream and if one was determined to be outside the user-defined 'accept' colour-space then this would queue a delayed 'reject' signal that would fire ejectors to take the nut out as it passed by them (typically with a compressed air jet in high volume machines). This principle of operation continues today though the algorithms have become more sophisticated and can now also consider the size & shape of the object as well as size of defects detected on the object.

This machine goes a step further. Starting with a high resolution image(s) it classifies each object & defect into type, measures their level of severity and from there makes an accept/reject decision. Perhaps the easiest way to explain how the system works it is to go through the process steps using a reject nut as an example.

The following two screenshots show the processing of a single frame of image data for reject & accept nuts.

(Click images to enlarge)

The system uses five such frames per nut in different orientations in order to cover the whole surface. Looking at the reject nut picture, the left half shows the unprocessed image and the result of the analysis is shown in the right half. The boundary of the nut is shown in green and defects are highlighted in red.

Frame Analysis

The bottom half of the images contain a table of the resulting analysis.

(Click to enlarge)

Line 2 of the tables is concerned with the overall nut, it takes a number of measurements of size, shape & colour and then in the final three columns determines that the reject nut is likely 'Rotten' with fairly high certainty and moderate severity.
Lines 4 thru 9 do a similar analysis with each of the identified defects, in this case they are mostly determined to be 'Fruit Spotting Bug' with varying levels of severity (one is misidentified as 'Mac Nut Borer' and indeed these can have similar appearance in some cases).

It is important to note that at this stage the user can not adjust the settings for the analysis algorithms. Also note that at this stage there has been no decision about what to do with the nut, rather it is simply measurements of severity of various faults. This is by design so that the machines can be calibrated such that all machines will output exactly the same results at this point in the analysis. This allows data from different machines (and different years etc) to be directly compared.

Decision Making

The next table shows the fusing of data from five image frames along with other sensors such as load cells as required. This is the basis for the final decision making process.

(Click image to enlarge)

Lines 6 thru 8 presents the data from the analysis of all possible defect types from the five image frames is presented as

Count - the estimated total number of occurrences on the whole surface of the nut. For example in the FSB (Fruit Spotting Bug) column it estimates there are 8.4 lesions which is probably quite accurate given that you can see approx seven on one side of the nut. The Rot (rotten) count of 1.0 means that it was classified such in every frame.
Peak - this is the severity value of the worst occurrence - for FSB it is saying the worst lesion scored 2545.
Avg - is the average severity of all observed occurences - for FSB it gives a value of 776.

In this case the principle defect on this nut is FSB but the table shows that the system has misidentified a few cases as MNB (Mac Nut Borer). There's also a few marks classified as Mechanical Damage and Germination - the latter may be valid as a split in the suture can be observed at the top of the nut image.

Values at this point still can not be adjusted with user settings, user control occurs at the next step.

Beside each of the calculated values is a code indicating quality. These codes are completely user definable, in this case it uses seven codes - P1, P2 are graded Premium; C1, C2 are graded Commercial; and R1, R2, R3 are graded Reject. How each calculated severity value in the table is converted to a code is also defined by the user and can be set up for each individual defect type.

Once all of the values have been coded the system assigns the nut the worst code observed - in this case the nut is marked Reject 3. Then there is a second user defined process that determines which of the five outlets nuts of that classification is sent to - in this case Outlet 3 (of five).

Batch Summary

(Screenshot incoming)

All of the data - images, sensors, analysis, decisions - can be dumped into data files for further statistical analysis. While this isn't really important for high volume sorting applications it is useful in scanning situations to be able to log all data, for example if working with a multi-year trial it would allow reprocessing of older raw data as improved algorithms and settings became available.

Conclusion

The system isolates the process of defect assessment from separation decisions by design. The defect assessment uses built in settings that can not be adjusted by the user whereas the separation decisions are controlled by the user. It sounds a little complicated but not as bad as it sounds, and it has the following advantages

Summary data from calibrated and invariable defect assessments can be compared across machines and across years etc.
This calibrated output is useful for lab/research applications and in commercial sorting.
Allows separation decisions to based on levels of defect 'severity' rather than abstract concepts such as colour spaces etc.
Highly flexible separation configuration.
Buyers of NIS can release their preferred separation configuration files that can be used with any machine thereby ensuring the output product conforms with their purchase specifications. A buyer might have several configurations tailored for different customers.

Further Work

It is good to have the machine actually sorting nuts and doing a good enough job to be useful but there is still a bit more to do. Finalizing the calibration and settings for the classification & severity assessment algorithms will take a couple of months. I will also be installing it into our process line in the next couple of weeks and putting it to work in real world conditions.

I am also planning to make some changes that will cater for multi-lane versions being run from a single control system in order to get throughput volumes up to commercial levels.

Robotic Framework

This is the first machine using a robotic framework that I have been developing for some time. The framework consists of three parts -

class libraries for multi-threaded PC based high level processing including vision analysis
standardized embedded classes running on (custom) general purpose PCBs that do machine control (you can see a stack of three of these prototype boards on top of the machine)
middleware classes on the PC that manage the link between non-deterministic classes on the PC vs the real-time classes running on the PCBs.

At this stage the framework seems to be doing as it's told with good efficiency though it's still under active development. Applying it to this sorter has highlighted a few areas that can be improved, in particular I will be making some modifications that would simplify using it with multi-lane machines.

The next application for the framework will be adding autonomy to the drone harvester, hopefully later this year.

April 2018 - Initial Accuracy Testing

I have started calibrating and characterising the machine recently. The following table lists the traits I am aiming for and their current status. These figures are still a bit rubbery as I'm still going through the process so these will be updated from time to time as new data becomes available. Stats in these tables is based on P=0.05, N=20.

Raw Measurement Accuracy

This first table shows the error for the various raw measurements the machine makes.

Parameter	Measure Unit	Measure Error (+/-) Single Nut	Measure Error (+/-) 20 Nut Batch Average*	Comment
Whole Nut - Colour	8 bit RGB	2.5 (2%)	0.5	Needs transform to standard colour scale
Whole Nut - Size Average Diameter Minimum Diameter Maximum Diameter Min Enclosing Diam	mm mm mm mm	0.20 0.50 0.50 0.30	0.05 0.15 0.15 0.10	= required slot opening for separation = required circle opening for separation
Whole Nut - Mass Whole Nut - Moisture Mass	gr gr	0.15 0.08	0.05 0.02	+/-0.1gr is possible in a half-speed high resolution mode. Experimental
Whole Nut - Calculated Metrics Volume Specific Gravity Shape Moisture Content	cm^3 gr/cm^3 mm/mm w/n	0.25 0.05 0.02 2%	0.05 0.02 0.01 0.5%	Volume / Mass Min/Max, Prolateness, Convex Deviation Experimental

*Indicative of single source batch. Depends on the in-batch variability, some scenarios are higher than others.

Classification & Severity Accuracy

This tables shows the accuracy for various high level categories of defect.

Category	% False Positives	% False Negatives	Severity Error	Comment
Whole Nut - Categories Normal Immature Sun Faded Aged/Rotten	<1%* - - <1%	<1%* - - <1%	5% - - 5%	* these can increase as other categories are added calibrating calibrating Age Faded & Rotten mix a bit
Defect - Categories				calibrating

* Perfectly 'Normal' nuts are Severity = 0

Rapid KR

The machine is also capable of doing a Rapid Kernel Recovery test, where either wet or dry nuts are cracked and placed in the machine, and KR is calculated a few seconds later. This can be done as additional function of the sorter with minor modifications, or a compact single purpose version can be built for in-field use.

This table shows accuracy for five NIS traits that the Rapid KR mode can measure.

Trait	Single Nut Accuracy	In-Tree Variability	20 Nut Batch Accuracy*	100 Nut Batch Accuracy*
Nut Diameter (mm)	0.6	3.5	0.8	0.4
Nut Mass (gr)	0.5	3.0	0.7	0.3
Kernel Diameter (mm)	0.5	2.7	0.6	0.3
Kernel Mass (gr)	0.2	1.0	0.2	0.1
Kernel Recovery	1.5%	4.4%	1.0%	0.5%

* There is substantial in-tree variability for these traits and it is the dominant source of error in the batch values. It can be ameliorated by increasing sample size. It is worth noting that these are roughly the same values you would expect from the standard 48hr test because it is also affected by the in-tree variability.

Visual Separation of Varieties

Finally the machine can also separate between different varieties of nuts based on the NIS appearance.

This table shows the separation accuracy for four common varieties.

Mixing From\To	A4	A16	A38	H246	Overall Error
A4	-	< 1%	< 1%	< 1%	1%
A16	< 1%	-	2%	11%	10%
A38	< 1%	2%	-	5%	7%
H246	< 1%	14%	4%	-	11%

This is very much a work in progress at this stage, I will be looking at a larger group of varieties in the near future.

May 2017 - System is Up!

So here is a video of the beast working, filmed in glorious shakey-cam.

Back To Home