Knowledge-based supervision and control of bioprocess with a machine vision-based sensing system

~ . . ,,.:,~..-..~, ~ :' ! ~ :. ~: ..,~ ,; 5~.~:~ ,.~ill~ ~ { ~ : , , :: ~:- .(m~[

E L S E V I E R Journal of Biotechnology 36 (1994) 25-34

journal of blotechnology

Knowledge-based supervision and control of bioprocess with a machine vision-based sensing system

Jinliang Ren, J.F. Reid *, J.B. Litchfield Department of Agricultural Engineering, University of Illinois, 1304 W. Pennsylvania Avenue, Urbana, IL 61801, USA

Received 15 January 1993; revision accepted 20 March 1994

Abstract

A computer supervised bioprocess system with an on-line machine vision sensing system was designed and implemented. The system consisted of a central process controller, a bench scale bioreactor, a computer-controlled automatic sampling system and a machine vision sensing system. A digital control unit, part of the bioreactor, maintained the process variables. The vision sensing system and the associated automatic sampling system were integrated with the bioprocess system to provide on-line cellular information, correlated to the cell concentration, cell growth rate, and cellular events. The sampling system delivered the sample to the microscope stage for the vision system to acquire images.

The central process controller (CPC) supervised all other subsystems via digital communications. The CPC directed the sampling system to take samples at defined time intervals or by special request. Images of samples were processed by the vision system afterwards and feature information was saved. Control decisions were made by the knowledge-based supervisory controller in terms of the processing variable setpoint adjustment. This bioprocess system could be used as a research technique. It could also be applied for quality and process control in industry.

Key words: Supervised control; Bioprocess automation; Machine vision; Automatic sampling

1. Introduction

The increasing impor tance of producing microbial and cellular products has led to an ever-

* This work was funded by and patent/licensing right are owned by the Biotechnology Research and Development Cor- poration, Peoria, IL. Contact the author or BRDC for information about licensing the use of this technology. Mention of trade or company names does not constitute endorsement by the author or the University of Illinois.

* Corresponding author.

increasing demand for cultivation methods to grow microbial, animal, and plant cells in an artificially crea ted envi ronment (Wang, 1986; Nyeste et al., 1981). To achieve a bet ter under- s tanding of these processes and to allow for making decisions regarding the best opera t ing condi- tions, process moni tor ing and control has become essential for bioprocesses (Balzer et al., 1984; Guthke et al., 1990; Mathlouthi , 1986; Titus et al., 1984).

Bioprocess variables are obta ined by biosen- sors and inst ruments for bioprocess control (P6cs

0168-1656/94/$07.00 © 1994 Elsevier Science B.V. All rights reserved SSDI 0168-1656(94)00053-F

26 J. Ren et al. /Journal of Biotechnology 36 (1994) 25-34

et al., 1988). A number of variables are taken together to constitute a basic set for monitoring a bioprocess (Armiger, 1985; Bull, 1985). Addi- tional variables can be measured according to the needs of a particular situation. For most applications, temperature, agitation rate, aeration rate, dissolved oxygen activity, pressure in the head space of the bioreactor, pH, foaming are sensed and controlled (Karim and Traugh, 1987; Fastert, 1990).

There are three particularly difficult problems in controlling bioprocesses due to the fact that a living system is involved (Rolf and Lim, 1985). First, there is a lack of appropriate instrumentation. Next, bioprocesses involve a large number of complex and highly interacting biochemical reac- tions and transport phenomena. Finally, the microorganisms themselves have a complex regula- tory system within the cells and an external control system can only manipulate the extracellular environment in hopes of affecting the intracellu- lar mechanisms.

A key limitation for complete bioprocess monitoring is the inability to measure many key parameters such as cell concentration and various substrate and production concentrations accu- rately, quickly, and on-line. A few commercial on-line sensors are available for substrate and product measurement in growth medium (Wang, 1986). On-line sensors for measuring the intracel- lular events of microorganisms which dictate the outcome of the bioprocess are not commercially available. Most of the commercial chemical- or biochemical-based analyzers are inadequate to be used for on-line analysis because a suitable interface device is not available.

Current methods for determining the status of the process require removal of samples from the bioreactor. Sample examination and assay provide information about the state of the process for the operator to make control decisions. Sub- sequent actions, such as changing the set-points of the processing variables, will be taken to make adjustments. Although periodic sampling is te- dious, it is still commonly used by industry for process monitoring (Wang, 1986).

Video microscopy has been used for off-line counting and classifying microscopic objects (Cos-

tello and Monk, 1985; Thurston et al., 1986). The information from microscopy is very important and useful for control of some bioprocesses. However, applications of video microscopy in full process automation are limited.

The objectives of this research were to design an automatic bioprocess control system by using a machine vision sensor along with other subsystems.

2. Materials and methods

2.1. System structure

The overall design of our automatic bioprocess system includes a central process controller, a traditional bioreactor with a local controller, a computer-controlled sampling system, and a machine vision sensing system (Fig. 1). This configu- ration can be recognized as a centralized process control system with local loop controllers. The Digital Control Unit (DCU; B. Braun Biotech, Inc.) controls the process through a microcomputer-based controller which mainly regulates the primary processing parameters through its built-in PID control modules.

The central process controller (CPC) supervises all other subsystems through digital communications. The CPC informs the sampling system to take samples at pre-defined or user specified time intervals. The sampling system prepares and delivers microliter samples of growth medium to a microscope. When the sample has been delivered to a viewing stage, the CPC activates a machine vision system to acquire images of the sample. The machine vision system processes images to extract image features related to the characteristic changes in the bioprocess. A feature database generated by the machine vision system is provided to the CPC to use for making control decisions. Control signals are formed and fed back to the DCU control module to make specified adjustments in the control of the process. Decisions on process adjustment are developed from specific bioprocess objectives to dy- namically optimize the process.

J. Ren et aL /Journal of Biotechnology 36 (1994) 25-34 27

DCU ~ CABLE

CONTROL BOX MICROSCOPE

R3-232

CONTROLLER

mm ~1--~ ~ VISION SYSTEM

Fig. 1. The automatic process system including central process controller, bioreactor, computer controlled sampling system and machine vision sensing system.

2.2. Automatic sampling and image acquisition

The automatic sampling system contains a sampling controller, a sampling operation device and a sample viewing station. The controller starts to perform sampling actions after receiving instructions from the CPC. It controls the operations of sample preparation and delivery, mixture of additives, and system cleaning. A typical sequence of the operations is as follows: 1. extract a sample from the bioreactor, 2. mix the sample with specified diluent or add

any chemicals or dye to enhance the sample visual properties,

3. deliver the prepared sample to the viewing stage of the microscope,

4. clean the sampling system. The primary job performed by the sampling

system is to deliver sample solution from bioreactor automatically to the microscope stage upon request of CPC. The sampling system also has a capability of diluting the sample in order to control the number of microscopic features which appear in images. Images acquired from samples with appropriate concentration will simplify the image processing because it is difficult for vision system to process images with a high concentration of cells.

The key element of the sampling operation device is a valve block which has 23 valves. Each valve can be individually opened or closed. There are tubes connected to the bioreactor, diluent containers, a vacuum source, a pressure source, and tubing connected to the microscope stage. Inside the valve block, there are pathways for liquid flow through, and a mixing chamber.

A portable computer (PC) was used as the sampling controller. The PC communicates with CPC via RS-232 interface and uses its parallel port to control the sampling operation device. The sampling controller was designed to preform the sampling sequences according to the instructions received from CPC. Five main sequences were designed to accomplish the tasks of drawing sample into the chamber, drawing diluent into the sample, bubbling the mixing chamber, deliver- ing sample to the microscope stage, and cleaning the sampling system.

To present samples to the microscope stage, a sample viewing station was designed which has a micro channel pathway for sample solution to flow through (Fig. 2). The sample viewing station was mounted right underneath the lens of the microscope. The sizes of the micro channel are critical, especially the depth of the channel. Prac- tically, we choose a micro-channel 20/xm (D) by


MICRO CHANNEL

Fig. 2. Construction of the sample viewing station used for viewing samples under the microscope.

20/zm (W) in cross-section for small cells. Differ- ent widths could be selected based on the size of the particles passing through the system. During sampling, images are acquired after each time of sample delivery. The images are saved to the hard disk for later processing. The image file name is based on the sample number and the image number for that sample.

2.3. Machine vision system

Machine vision was integrated as a sensor for detecting substrate and cellular properties. Sam- ple liquid extracted from the bioreactor was pre- sented to a microscope stage and microscopic images were captured by the camera of the vision system. Different microscope systems can be selected for simplifying detection of the properties of the growth medium. Machine vision sensing can sense two types of characteristics from the images; morphometric and photometric changes in the images of the growth medium. The machine vision system receives commands from the central process controller to perform image analysis. A series of operations are then executed to extract meaningful information from the images related to process control. The machine vision system: 1. acquires images during sampling, 2. executes the appropriate machine vision algo-

rithms to perform image processing for feature enhancement,

3. performs feature extraction and classification, and

4. returns image feature database to the CPC. Different bioprocess require specific se-

quences of operations for image processing. A flexible software framework was development to provide a range of capabilities which can be adapted to specific processes.

The machine vision system consisted of a microscope equipped with a video camera and an image processing system. In our study, a phase- contrast microscope (OLYMPUS BH2-RFCA) was selected. A CCD-72 series monochrome camera was selected from DAGE-MTI, Inc. The image processing system included several circuitry boards from Datacube and Vision System, Inc. for monochrome imaging.

2. 4. Central process controller

The central process controller is a supervisory system which gathers information from other controllers and supervises their operations. The CPC interacts with the sampling system, the bioreactor control system (DCU), and the machine vision sensing system. The primary functions of the central process controller are to: 1. activate the sampling system and receive infor-

mation on the status of sampling, 2. start the vision system for sensing the process

state and receive data about detected image features (such as average cell size and cell concentration),

3. read the status of the bioreactor controller (DCU),

4. make control decisions, using the knowledge base and current process state,

5. modify the setpoints and carry out the control. The CPC communicates with the DCU and

sampling system through an RS-232 serial line for both receiving information and sending control instructions. A Sun SPARCstation 1 + (Sun Mi- crosystems) was selected as the performing unit of the CPC due to its fast processing speed and multi-tasking feature. The Sun workstation com- municated with the DCU through an RS-232 serial link. The sampling subsystem was also linked to the CPC via an RS-232 serial communication line. The workstation was arranged to work as a primary processor for the machine vision

J. Ren et al. / Journal of Biotechnology 36 (1994) 25-34 29

system, too. The machine vision system was connected to the CPC through a VME bus extension connector. A graphical interface on the CPC provided information on the status of the CPC and the other subsystems.

2.5. System control software

The system software consisted of two pack- ages. One was a communication package with low-level and high-level functions in the form of C libraries. One library was for the DCU (DCULIB) and another for the sampling system (SAMPLIB). The low-level functions are shared by DCU and sampling system, and are located in the DCU library. Another software package con- sists of an interface program, control programs and database files. The interface program is mainly a working cell and runs under the X- Windows environment. A user can start all the activities provided from the main menu.

The control programs consist of a group of task subroutines, independent control programs and image processing programs. The task subroutines are for data collection, sampling and image acquisition. A character oriented protocol using an RS-232 interfac6 was used for both communications to DCU and sampling system following the telegram format proposed by B. Braun Biotech, Inc.

Our interface program was developed using Xt-Toolkit and OLIT (Open Look Intrinsics Toolkit) Widget Set (Sun Microsystems). The control software features in pull-down menu, graphical display, low-level control, individual subsystem (bioreactor controller and sampling controller) checking, flexible setting for different bioprocess and supervisory control. There are four main menus, and the submenu are 'popped- up' when the main menu icon is selected. Fig. 3 shows the structure of the menu tree. There are four submenus under the main menu TASK. SYS CONTROL start or stop running system

control program DATA COLLECT start or stop running data

collection program SAMPLING running sampling EXIT quit and exit system

--~DATA COLLECT ~ SAMPLING SYS CONTROL EXIT

2 5 I 0 S I t.Oh 6osI I ~.~,, 120 S I 2.0 h 3 0 0 S 12.5h

OTHER I OTHER

~ I L°~-~°- ~ I

_L__ I~TA II,~'~RVN. ~ I ~ ~ m ~ Vl ~ ~Ec,~t ~1 ~ PEESS v / ~ . o . ~ v ~ ~ o ,~ ~ ~

~ ~ l ~ v H ~ ~ c u ~11 ~ ~ Ico ~H ~ ~ ~ o~HE. i ~ I0o v p

Fig. 3. The menu tree structure of the control software.

Several utility functions were written to enhance the capabilities of the system software. A sampling subroutine communicates with sampling system to control sampling operations. Inside the sampling function, the ACQUIRE_IMG( ) function is called to acquire images. The number of images can be defined in the system setting menu. The ACQUIRE_IMG( ) function interacts with vision system and captures images and saves them to the hard disk until future processing is requested.

The control programs contain a main control subroutine which is assigned as a work procedure in the X-Windows interface program, and several external programs for control decision making and image processing. The purpose of this type of software arrangement is to make the software flexible and adaptable to different applications and control tasks. The flow chart of the main program is shown in Fig. 4. Based on flag set- tings, the program performs tasks in sequence. The system may be used for system control with data collection or just for data collection. Task sampling was designed for sampling system checking and image acquisition.

When the control program flag is set, the sampling action starts at defined time intervals. Sample images can be saved if requested. The ANALYSIS procedure begins at the end of sam- piing. The CPC first updates the decision support database CONTROL.DSP with current DCU


Fig. 4. The flow chart of the main program of the system control software.

the control system initialization, the defined process properties are set to these names so that the specific programs for image processing and process control will be called during the process. These programs are external, therefore allowing users to develop their own image process programs and process control programs without modifying and recompiling the whole system software.

A system testing was carried out to test the performance of the supervisory control system software. The test was performed by feeding central process controller with planned process pro- files and control information using cell-counts measured by machine vision to control process variables. A simulated process was designed to control the temperature and stir rate based on the average cell-count of the images acquired by the vision system during the testing. Yeast cells were selected for samples.

process values. Then a user-defined external image process program is called to process the saved sample images, and the image feature data are saved in the CONTROL.DSP file. The last stage of the ANALYSIS is to call the external control decision program to make decisions based on the current DCU process values and image information. The control decision mainly deter- mines that if the current DCU setpoint needs to be adjusted based on the pre-acquired process knowledge. Currently, the DCU setpoints for temperature, stir rate, pH and pO e are arranged as the control variables. The value for the setpoint will be reset at the end of the ANALYSIS procedure by sending the DATA_ SET command to the DCU.

The decision support database contains three kinds of data. The first kind is the previous process values and current process values. The sec- ond is the image feature data. This can be any image feature defined by user. The third is the new DCU setpoints from the decision making program. There are four setpoints arranged in order of temperature, stir rate, pH and pO 2.

The image processing program is specific to a defined bioprocess. Similarly, the decision making program is specific to the bioprocess. During

3. Method for system testing

The system input was cell-count, and temperature and stir rate were outputs. According to the cell-count level changes, the temperature and stir rate would change, respectively, at three distinct levels (Fig. 5). The temperature changed within the range of 26 to 34°C, and the stir rate changed

.................... I ................... $ , _ _ °:t 9 ~

0

.................... J ... . . . . . . . . . . . . . i ...................

~0 4~0 6'0 dO 160 Time (rninut~)

I--cell-count .......... temp - - s t i r rate I

1800

-1600

-1400

-1200 ~"

-~000 ~ I~ -800 ~-

-6017 ~

-400

-2013

0 120

Fig. 5. The designed course of testing using automatic sampling system. The stir rate and temperature will follow the changes of the cell-count according to the designed control rules.

J. Ren et al. /Journal of Biotechnology 36 (1994) 25-34 31

from 200 to 400 rpm. This values are arbitrarily selected to test the system and the software. The process control was based on the following rules: rule 1: IF (CELL COUNT IS BETWEEN

LEVEL 1 AND LEVEL 2) THEN (TEMP = 30)

rule 2: IF (CELL COUNT > LEVEL 2) THEN (TEMP = 26)

rule 3: IF (CELL COUNT < LEVEL 1) THEN (TEMP = 34)

rule 4: IF (ABS(SET T E M P - MEASURED TEMP) > = 3) THEN (STIR RATE = 400) ELSE (STIR RATE = 200)

The sample solution was prepared by dissolv- ing dry yeast into clean water. Three different concentrations were named sample A, B, and C. The quantity of yeast and water for those solutions are listed in Table 1. The samples were stirred for about 5 min and then remained in the glass test tube for several minutes. The top por- tion of the sample was used for testing.

The ceil-count was obtained from the image processing program of the machine vision system. The program processes an image in the image frame buffer and generates an output file in text format which lists the number of objects found in the image. For each sample, eight images were collected to calculate the average cell count. The previous study shows that more than five images must be used to compute the reliable average cell count (Richburg, 1992). The control program made decisions based on the measured cell-count. The samples with three different concentrations provided the input disturbances when the samples were switched during the process. The inlet tube of the sampling system was manually placed into another sample at a defined time.

Table 1 Yeast sample solutions used for the experiment with auto- sampling

Sample

A 13 C

Dry yeast (g) 0.1 0.2 0.4

Water (ml) 30 30 30

Table 2 Average cell-count computed for three samples

Sample Image AVG STD

1 2 3 4 5

A 14 18 7 12 17 13.6 3.93 B 27 22 26 26 25 25.5 1.72 C 41 57 60 54 59 54.2 6.91

The cell-count for these three prepared samples must be obtained so that the margins (Level 1, Level 2) can be determined and provided to the control program before the testing started so that the control program could detect the changes of samples. To obtain the cell-count, the prepared sample solution was injected through the sample viewing station using a syringe, and five images were collected for each sample A, B, and C. The sample viewing station was cleaned between samples. The average cell-count was calcu- lated for each sample (Table 2). Level 1 and Level 2 were set to 20 and 40, respectively.

The sampling interval was set to 6 min. Three times of sampling were performed for each sample that means the sample would be switched with one of the others after 18 min. The sample was drawn into the mixing chamber and delivered to the sample viewing station. A 40 × phase contrast lens was used in the microscope for the testing and the microchannel size of the sample viewing station was 0.02 × 0.2 mm. The schedule for samples is listed below: process time sample

0 min sample B 18 min sample C 36 min sample B 54 min sample A 72 min sample B 90 min sample C

Based on this schedule, the cell-count is expected to vary with time as shown in Fig. 5. The temperature is designed to vary with cell-count accordingly, but in the opposite direction. The stir rate will vary according to the temperature changes. The testing was run for nearly 2 h. The initial setting of the DCU was temperature to 30°C and the stir rate to 200 rpm.


4. Results and discussion of system testing

Fig. 6 is an example of the captured image. The result gives very accurate count for object in the image. The recorded process data are plotted

in Fig. 7. Since the temperature and stir rate were recorded at the same sampling frequency as cell-count, the plotted curves looked different from the designed. Most computed cell-count stayed in the defined range even though they had

Fig. 6. An example of the captured image of yeast sample.

J. Ren et aL /Journal of Biotechnology 36 (1994) 25-34 33

~ 60 (.) o~

~ 50. I~

~.~ 413- -~

~ ~0-

~ 20'

~ ]0"

~ 0 ~o ~o ~o ~o ~6o

Time (minute)

1800

-1600

-1400

-1200 "~

- 1 ~ ~

- ~ ~ - ~ ~

-41~

-~00

0 ~

I - - cel l -count . . . . t e m p - - stir rate I

Fig. 7. The recorded process values of cell-count, temperature and stir rate from the control experiment using automatic sampling. The stir rate and temperature were changed following the control rules with the detection of current cell count.

a larger variation compared to the results listed in Table 2. The temperature changed slowly as expected. The shape of the temperature curve followed the desired pattern, and correctly changed with computed cell-counts. The changing pattern for stir rate was more like the designed pattern. The stir rate raised to 400 rpm when the temperature had a sharp change. The last computed cell-count fell outside the designed region.

The results show that there was a large variation of cell-count for samples. However, all the variables followed the desired pattern. All the system components worked as desired under the supervisory control for complete process control and automation.

5. Conclusions

A knowledge-based control and supervision bioprocess system with a vision-based sensing system was developed. The system components included a traditional bioreactor with a local controller, an automatic sampling system, a machine vision microscopy system, and a central process controller. The system acquired cellular information of the process on-line to achieve the corn-

plete process control. Specific conclusions are itemized below. 1. The implementation of the automatic bioprocess system was completed. The interface modules for each subsystem was developed. The communication between central process controller and bioreactor controller (DCU) used an RS-232 serial interface. The same interface was used for communications between central process controller and sampling controller. The vision system and central process controller used the same computer (Sun Workstation). Some subroutine functions and independent task programs were developed for image processing and control. 2. The automatic sampling system was developed which could deliver microliter sample of growth medium to the microscope stage. The system was built with automatic dilution capability. The vision system was able to process images and gen- erate feature database on-line. 3. The central process controller was developed with X-Windows as a working environment. Be- side the normal data acquisition and process monitoring, the system software lets users set process recipes at any stages, define specific image processing procedures and define process control strategies. By using measured process variables plus the cellular information obtained from sampling and vision system, complete bioprocess control and automation was established.

References

Armiger, W.B. (1985) Instrumentation for monitoring and controlling bioreactors. In: M. Moo-Yong (Ed.), Compre- hensive Biotechnology: the Principles, Applications, and Regulations of Biotechnology in Industry, Agriculture, and Machine. Pergamon Press, Oxford, pp. 133-148.

Balzer, B., Hesse, J. and Schmidt, A. (1984) Automatic computer control of fermentation processes. Acta Biotechnol. 4(1), 11-16.

Bull, D.N. (1985) Instrumentation for fermentation process control. In: M. Moo-Yong (Ed.), Comprehensive Biotech- nology: the Principles, Applications, and Regulations of Biotechnology in Industry, Agriculture, and Machine. Pergamon Press, Oxford. pp. 149-163.

Costello, P.J. and Monk, P.L. (1985) Image analysis method for the rapid counting of Saccharomyces cerevisiae cells. Appl. Environ. Microbiol. 4, 863-866.

Fastert, H.J. (1990) Supervisory computer system. In: Daniel


R. Omstead (Ed.), Computer Control of Fermentation Processes, CRC Press, Inc., Boca Raton, FL, pp. 147-163.

Guthke, R., Schulz, V., Knorre, W.A., Haupt, B. and Wernst- edt, J. (1990) FERM: a decision support system for fermentation processes. Acta Biotechnol. 10(1), 63-71.

Karim, M.N. and Traugh, G. (1987) Data acquisition and control of a continuous fermentation unit. J. Ind. Micro- biol. 2(5), 305-317.

Mathlouthi, M. (1986) Computer-controlled bioreactor: an example of direct digital control of an industrial food process. In: M. Renard and J.J. Bimbenet (Eds.), Auto- matic Control and Optimization of Food Processes. Else- vier Applied Science, London, pp. 347-355.

Nyeste, L., Szigeti, L., Veres, A., Pungor Jr., E., Kurucz, I. and Hollo, J. (1981) Automated fermentation equipment II: computer-fermentor system. Biotechnol. Bioengin. 23, 405-417.

P6cs, M., Szigeti, L., Nyeste, L. and Holl6, J. (1988) Analytical and automation problems of fermentation processes. In: M. Renard and J.J. Bimbenet (Eds.), Automatic Control and Optimization of Food Processes. Elsevier Applied Science, London, pp. 315-325.

Richburg, B.A. (1992) Machine Vision Microscopy as an On- Line Sensor for Bioprocesses. Unpublished Ph.D. thesis, Agricultural Engineering Department, University of Illi- nois at Urbana-Champaign, IL.

Rolf, M.J. and Lim, H.C. (1985) Systems for fermentation process control. In: M. Moo-Yong (Ed.), Comprehensive Biotechnology: the Principles, Applications, and Regula- tions of Biotechnology in Industry, Agriculture, and Ma- chine. Pergamon Press, Oxford, pp. 165-174.

Thurston, G., Jaggi, B. and Palcic, B. (1986) Cell motility measurements with an automated microscope system. Ex- perimental Cell Res. 165, 380-390.

Titus, J.A., Luli, G.W., Dekleva, M.L. and Strohl, W.R. (1984) Application of a microcomputer-based system to control and monitor bacterial growth. Appl. Environ. Microbiol. 47(2), 239-244.

Wang, H.Y. (1986) Bioinstrumentation and computer control of fermentation processes. In: A.L. Demain and N.A. Solomon (Eds.), Manual of Industrial Microbiology and Biotechnology, American Society for Microbiology, Wash- ington, DC, pp. 308-320.

Documents

Knowledge-based supervision and control of bioprocess with a machine vision-based sensing system