Computers and Electronics in Agriculture, 8 ( 1993 ) 117-127 117 Elsevier Science Publishers B.V., Amsterdam

A message system to integrate diverse programs and databases in a farm decision support system

D.J. Parsons and D.G. Randle AFRC Silsoe Research Institute, Wrest Park, Silsoe, Bedford, UK

(Accepted 26 May 1992)

ABSTRACT

Parsons, D.J. and Randle, D.G., 1993. A message system to integrate diverse programs and databases in a farm decision support system. Cornput. Electron. Agric., 8:117-127.

A data integration system required for agricultural decision support software has been analysed, and a model proposed using message passing with an intelligent kernel in a multi-tasking operating system. A trial system has been built using information storage and decision support programs suita- ble for dairy farms. These include a herd records database, a field events database, a ration formula- tion program and a grassland utilisation planning program.

1. INTRODUCTION

Researchers in several countries are working on the concept of fully inte- grated systems of decision support software for farms (Kuhlmann, 1990). One such project at the Silsoe Research Institute in England is known as the Information Technology Farm. The system needs to include data from a wide variety of sources, including data loggers such as milk recording or weather stations on the farm, local databases such as field or herd records, and remote databases for market information or weather forecasts. The project also in- cludes the development of new techniques to aid decision making using such information.

An analysis of the data integration system has been carried out and a small trial system has been built. The analysis led to the choice of a multi-tasking system in which different programs communicate by sending messages. This allows a standard form for data exchange to be defined without constraining the methods of data storage used. This contrasts with other approaches, such

Correspondence to: D.J. Parsons, AFRC Silsoe Research Institute, Wrest Park, Silsoe, Bedford MK45 4HS, UK

118 D.J. PARSONS AND D.G. RANDLE

as Harsh, Brook and Harmon (1990) and Sz6kely (1990), which integrate systems around common databases.

The trial system, for a dairy farm, contains a simple message system kernel, three databases, of which only one (the cow records) currently uses message passing, and several programs which use data from the databases. The herd database is a commercial program, not written for the system, so a server program was written to respond to messages requesting data. Similarly the ration formulation program requires a small 'shell' program to request its data.

This paper describes the concepts involved, the results of the analysis, the message system, and the server and shell programs.

2. MESSAGE SYSTEM

2.1 Concepts

One of the objectives of the IT Farm project is to provide a common method of access to data stored on the farm computer. This will allow any program written for the system to request data about fields, crops, livestock, weather and so on, without needing information about how or where the data are stored.

One way of providing this would be to define a complete database system for the whole farm, and store all data in it. This presents practical problems, given the present diversity of companies producing software for agriculture, because it precludes the use of existing farm database programs, which all have their own formats for storing data. Furthermore, it can not easily be extended to the next level of integration: the inclusion of data from other computers on the farm or beyond.

The use of a message passing system provides greater flexibility. When an application program needs to store or retrieve data, it issues a request and waits for the response, which might be an acknowledgement that the data have been stored, or the retrieved data, or part of a more complex sequence if a large volume of data is being transferred. Because the format of the mes- sages is standardized, the application programmer does not need to know how or where the data are stored; they might even be on a different computer. Database maintenance is separated from application programs, so the data- base system could be changed without affecting other programs.

Existing application programs needing data about the farm can be inte- grated by the use of specially written 'shell' programs. The shell program is able to request the data required by the application, run the application and pass data to it in the way it expects. This can be quite difficult, but is worth doing for selected programs. This will be made easier by writing a generalised shell which can be tailored to applications. It may also be possible to write a

DATA INTEGRATION SYSTEM FOR A FARM DECISION SUPPORT SYSTEM 1 19

trainable shell, which can be 'taught' that a program requires particular data items without the need for extra programming.

The system was analysed using the Yourdon Structured Method (Yourdon, 1989). This technique models the problem in terms of dataflows and state transitions, and produces models for different stages from the behavioural model to the implementat ion model. Several different implementations were possible, of which the simplest was a memory resident program or a device driver to handle requests. This was rejected as being too limited. Most of the other possibilities involved some form of multi-tasking. To reduce the devel- opment time a suitable, readily available, multi-tasking operating system was sought, which had to be capable of running existing MS-DOS software with as little performance degradation as possible on inexpensive computers. Of those available at the time, DESQview (Quarterdeck, 1987 ) was selected be- cause it met those requirements and provided for inter-process communica- tion by use of mailboxes. Because most of the analysis was independent of the implementation, it could easily be transferred to another operating system in the future, provided it had multi-tasking and inter-process communication.

2.2 Description and terminology

The method by which messages are used to request and transfer data is called Inter-process Message Protocol (IMP). A common destination for mes- sages is provided by the IMP-kernel. This is the first program started when DESQview has been loaded and must always be available. The kernel has a named mailbox, which can be located by any program needing to initiate an exchange of data. Such a program is called an IMP-client. The request is not normally processed by the kernel, instead it starts a separate program, called an IMP-server, which is responsible for the data being retrieved or stored. A typical system may contain many servers, one for each database, and the ker- nel identifies the correct one to use by means of a data dictionary. The data dictionary contains a unique identifier for each possible data item, informa- tion about its format, precision and so on, and details of the server responsi- ble for it. All messages in IMP use the definitions and formats given in the data dictionary. It is possible for a program to function as both a client and server.

When the kernel receives a reply from a server it has to know which client issued the request: in a multi-tasking system it is not safe to assume that there is only one client at any time. The client supplies this information by includ- ing the address of its mailbox in the original message. The server transfers this to the destination field of its reply, to be used by the kernel when for- warding the reply. The same method can be used to acknowledge requests to store data. Table 1 shows the structure of the messages used in the first ver- sion of IMP.

120 D.J. PARSONS AND D.G. RANDLE

TABLE 1

Message structure

Request

Bytes Contents Use

0-1 Steering field 1 Search definition 2-3 Steering field 2 Source code 4-5 Steering field 3 Access code 7-16 Undefined For future use

17-32 Search key 49-256 Data

257-259 Mail address Address of clinet's mailbox

Reply

Bytes Contents Use

0-15 Status Number of records, or error code 17-32 Search key Same as request 33-48 Date and time 49-256 Data May include address of server's mailbox

257-259 Mail address Address of client's mailbox.

Ifa large volume of data has to be transferred, for example a record for each animal in a herd, it becomes inefficient to route every message through the kernel, so a bypass mechanism is provided. The server, like the client, places its mailbox address in each message it sends. Thus, once a pair of messages has been exchanged, the client and server have each other's address and can communicate directly. The normal protocol when a client is retrieving data is for the first reply from the server to contain an acknowledgement and status information, such as the number of records to be transferred. The client then controls the transfer by sending an acknowledgement on receipt of each re- cord from the server, or by sending an instruction to halt the exchange. This is illustrated in Fig. 1. (Note that this is not a Yourdon dataflow diagram. ) A similar procedure may be used when storing data.

IMP clients and IMP servers are said to be IMP-aware. A program that is not IMP-aware may be made to use IMP by the provision of an IMP-shell, which is a program to interface between it and IMP. The shell may function as a client or a server depending on the program involved.

From the point of view of the user, IMP clients usually correspond to ap- plication programs which need to use data, for example mathematical models or expert systems. However, an application program may also function as a server, if another program requires its results. Most IMP servers correspond

DATA I N T E G R A T I O N SYSTEM FOR A FARM DECISION S U P P O R T SYSTEM 121

/

Ker ne I /

/ ? ",,%, / / ",, \

First / / / f i r s t First \,\ \ F i r s I r e q u e s t / / / / r eia i y r e q u e s t " \ "" \ r ela ~ >,'

/ I( '~ "

l i~l @ n t N e , t ~@r-'~ @r

J

Fig. 1. Data flows in IMP.

to databases. A database program may have IMP server functions built in, or a separate server may be provided. A separate server is usually smaller and simpler because it does not need a user interface.

A simplified version of the system has been written to demonstrate some of the principles involved. The data dictionary is not used and there is only one server, so the kernel does not have to analyse the message it receives. However the client-kernel-server and bypass routes both operate as de- scribed. Further details are given in Section 3.

3. AN APPLICATION TO A DAIRY FARM SYSTEM

The trial system contains several of the components needed for a dairy farm. These are a cow records database, a field events database, a meteorological database, a ration formulation program and a decision aid for planning graz- ing and silage-making.

3.1 Cow records

Cow records are stored using the Farmgate Dairy program from Farmgate Limited (Farmgate, 1990). The database contains data on 81 cows and heif- ers, plus a few followers. The data include monthly milk recordings, fertility records and much more. These are stored as indexed binary files.

3.2 Meteorological records

The meteorological data consists of daily recordings from Hurley in south- east England of several variables, principally air temperature, soil tempera- ture, rainfall, radiation and potential soil moisture deficit. Two sets are used:

122 D.J. PARSONS AND D.G. RANDLE

the 'current'year, actually data for 1989; and the historical data, which are the averages for 1961-88. These are stored as binary random-access files.

3.3 Field event records

A simple field event recording program was written to incorporate some novel features. The types of event recorded are: fertiliser application, cutting for silage, yield measurement, moving cows in or out. Each field may have only one record on any day, but there may be several events in the record. The program checks that the only records entered are for valid fields and dates. The program incorporates a semi-empirical model of grass growth driven by weather data, which estimates the amount of grass available and its quality (D-value, protein and metabolisable energy). Whenever a record of a silage cut or a yield estimate is entered, the yield given is compared with the predic- tion of model and used to adjust the model to reduce the error in future pre- dictions. Yield measurements can be assigned confidence levels, which con- trol how strongly they affect the model. This program also calculates silage production and records the quantity and quality in each silo. The data are stored in ASCII files.

3.4 Ration formulation

The maximum profit ration formulation program CAMDAIRY from Uni- versity of Sydney and New South Wales Department of Agriculture (Hulme et al., 1986) formulates rations for the system. This requires data on cow group sizes, calving dates, conception dates and so on, normally entered man- ually. These are used, together with feed prices, feed composition data, milk prices and ration constraints, in a linear program which finds the maximum profit ration for the herd at any time.

3.5 Grazing and silage planning

This decision support program was specifically written for the system. It takes the data on the field events to date and the planned silage cuts, then assigns grazing cows to fields not required for silage, and simulates grass growth and cow feeding to predict the intake of grass, silage and concentrates. The planned silage cuts may be adjusted interactively to attempt to find a plan that provides adequate grazing and winter feed, and to see the effect on pur- chased feeds. It can also optimize the cutting plan to minimize the cost of purchased feeds.

DATA INTEGRATION SYSTEM FOR A FARM DECISION SUPPORT SYSTEM 123

Field ~ / f Cow/herd recording ~ \,, recording

Fig. 2. Data flows between database programs (top), databases (middle) and decision support programs (bottom).

3.6 Interconnections

Figure 2 shows the data files used by each program. In principle the ration formulation program needs to use the field and silage databases for data about forage quality, but these links were omitted from the demonstration system for simplicity.

The cow records were chosen as the pilot study for IMP; all other data are read and written directly by the programs that use them. All requests for data from the cow records database go through the kernel to the database server. Because all the programs that use the cow data require the same information, a program was written to collect the data, summarise it, and store it in a sep- arate file. The ration formulation program has a shell program to read the data for it.

3. 7 Cow records database server

The cow records are stored in indexed binary files, and the Farmgate pro- gram is not IMP-aware, so a stand-alone server was written. It is written in the C language, and it contains definitions of the data structures used by Farmgate to store data, a database reading section, and a mailing system. The Farmgate database consists of a number of files, each of which has an index to the data records stored at the start of the file. The data definitions within the server program consist of structure statements defining the structure of the file index and of the records within each of the files.

124 D.J. PARSONS AND D.G. RANDLE

TABLE 2

Data items transferred in trial system

Item File

Cow number CLF Group number CLF Live weight CLF Lactation number CLF Dry cow? CLF Cow status CLF Days in milk CLF Projected calving index CLF Milk price CUMF Milk butterfat CUMF Milk protein CUMF Milk lactose CUMF Latest yield record CUMF Previous peak yield CUMF

CLF, Current lactation file. CUMF, Cumulative milk records file.

For simplicity a standard request is defined which requires data from the current lactation and cumulative recording files for a group of cows. The items transferred are listed in Table 2. When a data request is received the steering bytes are decoded, and the current lactation and cumulative data files are opened. Two bytes of the steering code are used to describe how the data is required; this can be either: 0 - Read all cows 1 - Read all group 1 cows 2 - Read all group 2 cows etc.

This information is used to set up a filter which will only allow data relating to cows in the particular group (or all cows) to be selected. Initially the in- dexes for the two files are read, sorted into cow number order, and merged to form a combined index. The data records are then read using the filter infor- mation and valid cow numbers are marked in the index and a count of valid cows generated. A request-received mail is then transmitted to the client via the kernel with the first four bytes of the status field set to the number of records to be transmitted, and the first four bytes of the data field set to the address of the server mailbox. This allows the kernel bypass mode described earlier to be used for the actual data transfer. Once this has been done the actual data records are sent, each record being made up from portions of the two database records associated with each cow, space filled to 208 bytes in

DATA INTEGRATION SYSTEM FOR A FARM DECISION SUPPORT SYSTEM 125

order to fill the mail message. The client program sends a data request mail directly to the server, using the mailbox address passed in the data bytes of the request-received mail referred to above. The server responds by sending the first data record in the data portion of an outgoing mail to the address contained in the request mail. This process is repeated until all the requested cow records have been sent.

In the trial system a fixed set of items is transferred from this database, as shown in Table 2. The next development will be to allow a program to request specific items of data. To do this a data dictionary is being constructed, which will give each item a unique identifier. Application programs will issue re- quests to the kernel containing the list of data dictionary items in the data field; the kernel will forward the request, and the server will decode the iden- tifiers, read the appropriate items from the appropriate files, and send the data back.

3.8 Feed groups data

As mentioned above a program (COWSTATE) was written to collect the data on the feeding groups in the herd. The main groups are: 1. Lactating cows not in calf 2. Lactating cows in calf 3. Dry cows in calf 90-99. Followers.

COWSTATE requests the data for the whole herd by sending a message to the kernel asking for group 0. The data for each group are accumulated, av- eraged and written to a file. Most of the data items required by CAMDAIRY and the other programs are directly available from the Farmgate database, namely live weight, days in milk, milk price, butterfat content, protein con- tent, lactose content and current yield. Heifers can be identified by the lacta- tion number, and the number of days in calf is derived from the days in milk and the projected calving index.

3.9 C A M S H E L L - Camdairy shell program

A client shell program, CAMSHELL, was written to transfer data from the Farmgate database to the CAMDAIRY ration formulation program.

CAMSHELL first starts COWSTATE as a separate process to create the group data file, and waits until it has finished. It then reads the file, using the same data structures as COWSTATE, and removes from the list all elements for followers, that is groups other than 1, 2 and 3. It then starts Camdairy as another process and pauses to give it t ime to load. CAMSHELL simulates a user by sending keystrokes to Camdairy via DESQview's key___swrite func-

126 D.J. PARSONS AND D.G. RANDLE

tion. It sends the keys needed to move through the menus, and sends each item of data by converting it to a text string and sending one character at a time, with pauses to allow Camdairy to catch up (and users to see what is happening). Finally CAMSHELL navigates to Camdairy's Formulate menu and unloads itself. This allows the user to press Enter to solve for the maxi- m u m profit ration, or Escape to go back and change other data.

4. CONCLUSIONS

A basic system of data level integration for farm software using message passing in a multi-tasking operating system has been developed and demon- strated. It allows existing programs and databases to be integrated by the use of shells and servers, and provides a method of access to data which is inde- pendent of the method of storage. Although the demonstrat ion system cur- rently only accesses one database and uses a single type of request, the design includes the use of a data dictionary, which will include information on the method of data retrieval to be used by the kernel. The client-kernel-server model allows complete flexibility for programs to request data while elimi- nating the need for the programmer to know how or where the data are stored.

The use of server programs also increases the flexibility of data storage. The server may, as in the demonstrat ion system, simply read the contents of a database. It could be a program that communicates with a data logger or an- other computer on the farm using the same message structure. For data on external factors such as market prices, it could communicate with an outside database. Finally, if the data were not available automatically from any source it could ask the user to enter data manually. The system therefore allows a great deal of flexibility without placing additional burdens on the writers of application programs.

REFERENCES

Farmgate, 1990. Farmgate Dairy System. Farmgate Computers, Corby. Harsh, S.B., Brook, R. and Harmon, R., 1990. AIMS Agricultural integrated management soft-

ware. In: F. Kuhlmann (Editor), Integrated Decision Support Systems in Agriculture: Suc- cessful practical applications. Proc. 3rd Int. Congr. Computer Technology, 27-30 May 1990, Frankfurt, Germany. Deutsche Landwirtschafts-Gesellschaft, Frankfurt, pp. 1-11.

Hulme, D.J., Kellaway, R.C. and Booth, P.J., 1986. The CAMDAIRY model for formulating and analysing dairy cow rations. Agric. Syst., 22: 81-108.

Kuhlmann, F. (Editor), 1990. Integrated Decision Support Systems in Agriculture: Successful practical applications. Proc. 3rd Int. Congr. Computer Technology, 27-30 May 1990, Frank- furt, Germany. Deutsche Landwirtschafts-Gesellschaft, Frankfurt, 328 pp.

Quarterdeck, 1987. DESQview. Quarterdeck Office Systems, Santa Monica, CA. Sz6kely, Cs., 1990. Decision orientated integration in the management information system of

large scale farms. In: F. Kuhlmann (Editor), Integrated Decision Support Systems in Agri-

DATA INTEGRATION SYSTEM FOR A FARM DECISION SUPPORT SYSTEM 127

culture: Successful practical applications. Proc. 3rd Int. Congress Computer Technology, 27- 30 May 1990, Frankfurt, Germany, Deutsche Landwirtschafts-Gesellschaft, Frankfurt, pp. 24-42.

Yourdon, E., 1989. Modern Structured Analysis. Prentice-Hall, Englewood Cliffs, N J, 672 pp.