Despite the discovery of new drugs and therapeutics for the treatment of diseases, confirmation of their effectiveness and safety of use requires additional long-term studies. A safe and fast method for this that does not require research on living organisms is computer molecular modeling. The application of artificial intelligence (AI) methods is also an important step today [1,2]. The most well-known and used molecular modeling methods are molecular dynamics and docking methods, and so on [3,4]. However, despite having several advantages, these methods also have disadvantages that can be of critical importance when conducting experiments.

We propose an effective approach to the modeling and study of biological systems and biochemical processes, particularly processes in cells under various scenarios of interaction: an algebraic approach that has been proven effective in many other scientific areas. In the modeling process, we combine an algebraic approach with AI methods.

The algebraic approach is based on the theory of agents and environments initiated by Letychevskyi and Gilbert [5]. This approach is implemented in our software tools for the modeling of biological models.

We have developed a formal model of the cell apoptosis process, which various scenarios of interactions between cell elements and different agents (enzymes, viruses, nanoparticles, etc.) can cause. When running a given model, we can analyze the changes in the environment under certain conditions and explore the properties or states of the environment that can be achieved in the modeling process.

In the developed models, we consider the following properties: (1) reachability of the cell degradation phase or apoptosis process inhibition under given influencing factors and their parameters (quantitative ratio between metalloproteinases/glutaminases and their inhibitors, the influence of nanoparticles and/or irradiation, etc.); and (2) determination of the initial state of the environment (indicators of the temperatures, concentrations, and structures of substances; acidity, characteristics of the studied cell, etc.) needed to reach the desired behavior scenario.

A feature of algebraic modeling, in contrast to simulation modeling and probabilistic methods, is its ability to abstract from specific values and consider multiple scenarios of system behavior rather than one specific scenario. This makes it possible to conduct an effective search for the environment or substances that have the necessary effect on cellular processes and that can be used to destroy diseased cells in the corresponding treatment process, particularly in oncological diseases.

We can consider modeling at different levels of abstraction depending on the chosen experiment: at the level of the atomic structure of substances and quantum–mechanical interaction, at the level of the molecular structure of substances, and at the level of interaction of biological objects. 

To effectively search models and narrow the search space for them due to the high complexity of models, neural networks are used, which are trained on known scenarios of cellular processes.

At this research stage, a method of applying algebraic modeling was developed to study the effect of enzymes such as multidomain zinc metalloproteinase, glutamine, and transglutaminase 2 on the activation and inhibition of the cell apoptosis process.