Résumé

ARCH-COMP 2021 Category Report: Falsification with Validation of Results

[Link to Paper] • January 22, 2022

Gidon Ernst, Paolo Arcaini, Ismail Bennani, Aniruddh Chandratre, Alexandre Donzé, Georgios Fainekos, Goran Frehse, Khouloud Gaaloul, Jun Inoue, Tanmay Khandait, Logan Mathesen, Claudio Menghi, Giulia Pedrielli, Marc Pouzet, Masaki Waga, Shakiba Yaghoubi, Yoriyuki Yamagata and Zhenya Zhang  •  8th International Workshop on Applied Verification of Continuous and Hybrid Systems (ARCH21)

This report presents the results from the 2021 friendly competition in the ARCH workshop for the falsification of temporal logic specifications over Cyber-Physical Systems. I worked on developing and presenting results from the CPS-Lab@ASU's team, S-TaLiRo, on various benchmarks including Automatic Transmission, Neural Networks, F16s, Chasing Cars and many more for various kinds of input signals.

Development using: MATLAB, Simulink, Stateflow, Python

Part-X: A Family of Stochastic Algorithms for Search-Based Test Generation with Probabilistic Guarantees

[Link to Paper] •  [Link to Code] •  October 20, 2021

Giulia Pedrielli, Tanmay Khandait, Surdeep Chotaliya, Quinn Thibeault, Hao Huang, Mauricio Castillo-Effen and Georgios Fainekos

Requirements driven search-based testing (also known as falsification) has proven to be a practical and effective method for discovering erroneous behaviors in Cyber-Physical Systems. Despite the constant improvements on the performance and applicability of falsification methods, they all are best-effort methods which do not provide any guarantees on the absence of erroneous behaviors (falsifiers) when the testing budget is exhausted.

Our proposed algorithm not only estimates (bounds) the probability that falsifying behaviors exist, but also it identifies the regions where these falsifying behaviors may occur. We demonstrate the applicability of our approach on standard benchmark functions from the optimization literature and on the F16 benchmark problem.

My contribution in this paper was end-to-end development and management of the entire Part-X toolbox, and it integration with the S-TaLiRo toolbox For this work, I worked on various implementation including gaussain processes and bayesian optimization from scratch. I also worked on developing results on various benchmarks including F16, Automatic Transmission, etc.

Technologies Used: Bayesian Optimization, Guassian Processes, Mathematical Optimization, Statistical Theory, Hypothesis testing, Formal Verification

Development using: Python (for Development), Poetry (for version control), Stateflow, Simulink.

Impact of news on the commodity market: Dataset and results

[Link to Paper] •  [Link to Dataset] •  October 20, 2021

Ankur Sinha and Tanmay Khandait  •  Advances in Information and Communication, Springer International Publishing (2021)

Over the last few years, machine learning based methods have been applied to extract information from news flow in the financial domain. However, this information has mostly been in the form of the financial sentiments contained in the news headlines, primarily for the stock prices. In our current work, we propose that various other dimensions of information can be extracted from news headlines, which will be of interest to investors, policy-makers and other practitioners. We propose a framework that extracts information such as past movements and expected directionality in prices, asset comparison and other general information that the news is referring to.

We develop a dataset of 11,412 new items, human-annotated news headlines (released with this study) collected from the period 2000–2019. We apply this framework to the commodity “Gold” and train the machine learning models on the dataset. We experiment to validate the causal effect of news flow on gold prices and observe that the information produced from our framework significantly impacts the future gold price.

Technologies used: Data Mining, SVM, RNN, LSTM, GRU, Time-Series Analysis, Hypothesis Testing, Natural Language Processing, Deep Learning

Development using: Python, R (for Development), AWS (for compute)

A Gradient-based Bilevel Optimization Approach for Tuning Hyperparameters in Machine Learning

[Link to Paper] •  July 21, 2020

Ankur Sinha, Tanmay Khandait and Raja Mohanty

The hyperparameter optimization problem is inherently a bilevel optimization task in nature, and there exist past studies that have attempted bilevel solution methodologies for solving this problem. In this paper, we propose a bilevel solution method for solving the hyperparameter optimization problem that does not suffer from the drawbacks of the earlier studies. The proposed method is general and can be easily applied to any class of machine learning algorithms.

We formulate the neural network hyperparameter tuning problem as a bilevel-optimization problem and solve it using gradient-based methods. The innovation is in terms of relaxation of strict assumptions, as opposed to previous work. The tuning process is sped up 3-4 times by using fewer lower level function calls than grid search and BO.

The entire pacakage was built from scratch including the implementation of neural networks, loss functions, and their gradient based optimization.

Technologies used: Hyperparameter Tuning, Bilevel Optimization, Neural Networks, Bayesian Optimization, Kriging model, Machine Learning

Development using: Python, MATLAB (for Development), AWS (for compute)

IOT based Adaptive Bilevel Switching System

[Under Review] •  September 2021

Kshitij Palghane, Heet Sankesara, Tanmay Khandait, Mandar Jatkar and Kamal K. Jha  •  Submitted to IEEE Systems Journal

Through this paper, we want to present an approach to build an automation system by proposing a switching mechanism that is easy to use as well as fundamentally different from the ones being used.

Most of the existing automation systems depends entirely on remote connections and have no reliable failure prevention mechanism. In this paper, we propose a bilateral switching mechanism that works remotely as well as manually. In the case of system failure, we have implemented a robust failure prevention architecture in place which will shift the entire system from remote model to an entire manual system.

Preliminary work for this paper was done during my Bachelor's Thesis.

Technologies used: Mesh Network, Embedded Systems, Electronics, Data Analysis, PCB Design

Development using: Python (for Software Development), C/C++ (For Software/Hardware Interface)

Bio-Inspired Optimization

CSE/IEE 598: Bio-Inspired AI and Optimization, Dr. Ted Pavlic, ASU

[Project Link] •  January 2022 - Present

This project is an ongoing project and currently contains implementation of Genetic Algorithms from scratch. Current implementations of multi-objective optimization using NSGA-II and simulated annealing are not available since the files are being graded by the instructor.

Development using: Python

Tags: Genetic algorithms, Simulated Annealing, Statistics, Optimization, Multi-Objective Optimization , NSGA-II

Kriging Based GPR with Nugget Effect

Graduate Research Assistant, ASU

[Project Link] •  December 2021 - Present

This project is an implementation of Kriging-based GPR with nugget Effect based on Dr. Giulia Pedrielli's MATLAB Implementation. This project was developed in order to aid the Part-X library, as well as stable use for various projects at the lab involving any research/work with GPR.

Although developed for internal usage, I plan to fix bugs and develop test-cases to come up with a stand-alone package for access to wider community.

Development using: Python. Scikit-learn

Tags: Linear Algebra, Global Optimization, Gaussian Process Regressor, Statistics

NNRepLayer - Neural Network Repair

Graduate Research Assistant, ASU

Other Contributors: Keyvan Majd

[Project Link] •  December 2021 - Present

Deployment of Deep Neural Networks (DNNs) in safety-critical settings requires developing new tools and techniques for the certification and analysis of such systems. NNRepLayer, an optimization-based tool in Python for the Deep Neural Network (DNN) Repair problem, solves the problem of modifying the weights of a single layer in a trained DNN to address unsafe behavior. NNRepLayer formulates this problem as a Mixed-Integer Quadratic Program (MIQP). Given the trained network, a collection of adversarial and correct sample, and target layer, NNRepLayer repairs the network to enforce the safety requirements on the network. Our tool supports output constraints represented in the form of linear and mixed-integer linear equalities\inequalities. NNRepLayer supports input architectures in both TensorFlow, and PyTorch, and also utilizes the well-known mixed-integer programming solvers including Gurobi and CPLEX.

My contribution was the development of entire structure of the toolbox, development of parsers for user-centric constraints, and running the experiments for the upcoming paper using our toolbox. Future work will also involve development of documentation/uder guide for the toolbox.

Development using: Python, Gurobi, Scikit-learn

Tags: Neural Network, Mixed Integer Quadratic Programming (MIQP), Safety, Safe ML, Toolbox

Part-X: A Family of Stochastic Algorithms for Search-Based Test Generation with Probabilistic Guarantees

Graduate Research Assistant, ASU

[Project Link] •  December 2021 - Present

Development of toolbox for the paper Part-X: A Family of Stochastic Algorithms for Search-Based Test Generation with Probabilistic Guarantees

My contribution was the development of entire structure of the toolbox, development of parsers for user-centric constraints, and running the experiments for the upcoming paper using our toolbox. Future work will also involve development of documentation/uder guide for the toolbox. Some key features include:

• Development of Core Structure - Trees, for identifying and classifying regions based on their probablity of falsification, while keeping a count of budget (maximum function calls).
• Implementation of Bayesian Optimization and Gaussian Process Regressors
• Development of toolbox for access to wider community
• Development of User guide

Development using: Python, Gurobi, Scikit-learn

Tags: Neural Network, Mixed Integer Quadratic Programming (MIQP), Safety, Safe ML, Toolbox

Using Verification Tools to Prove Safety of a NN-based Cyber Physical System

Safe Autonomy for CPS by Dr. Georgios Fainekos, ASU in Fall 2021

[Project Link] •  August 2021 - December 2021

Neural Networks are being increasingly used in the domain of perception, prediction and control of various cyber-physical systems (CPS). This has also led to an increased research in the verification and testing community to develop tools that can verify the properties of a neural network. This becomes extremely useful when deployed in systems which have a huge cost in case of a failure, aka, safety-critical systems. In this project, we utilise the ACAS Xu benchmark and try to find a single falsifying input that can falsify maximum number of neural networks from the dataset. I use the Part-X algorithm to find the falsifying input and compare it with random uniform sampling methods.

Development using: Python, Tensorflow

Tags: Neural Network, Mixed Integer Quadratic Programming (MIQP), Safety, Safe ML, Toolbox

Perception in Robotics Projects

Perception in Robotics by Dr. Yezhou Yang, ASU in Spring 2021

[Project Link] •  January 2021 - April 2021

1. Project 1:
   • Initials in ROS


2. Project 2a:
   • Pinhole camera model and calibration
   • Stereo Calibration and Rectification
   • Sparse Depth Triangulation
   • Dense Depth Triangulation


3. Project 2b
   • Planar Homography,
   • Camera Pose estimation and Reconstruction from Single View and Multiple Views


4. Project 3:
Pursuit Evasion Game:
• Development of custom environment in Gazebo
• Generation of map using SLAM
• Pursuit Evasion game where human is the evader and waffle bot is the pursuer.

Development using: Python, Tensorflow

Tags: Neural Network, Mixed Integer Quadratic Programming (MIQP), Safety, Safe ML, Toolbox

Graduate Research Assistant

School of Computing and Augmented Intelligence, Arizona State University  •  June 2021 – Present

• Worked in the field of Falsification of Cyber-Physical Systems to generate research papers, code base and research as deliverables to DARPA-ARCOS Program.
• Working on Neural Network Repair Problem by generating toolboxes and related research.

Research Associate

Indian Institute of Management Ahmedabad  •  August 2019 – November 2020

• Worked on research in the field of data-mining and news analytics for the commodity gold as recommendations to the Indian Gold Policy Center.

• Worked on research in the field of Bilevel Optimization to develop alternate approach for hyperparameter tuning of Neural Networks.

Arizona State University

Master of Science in Computer Science  •  4.0/4.0 CGPA  •  January 2021 – December 2022 (Expected)

Coursework: Planning/Learning Methods AI (by Dr. Siddharth Srivastava), Image Analytics & Informatics (by Dr. Jianming Liang), Perception in Robotics (by Dr. Yezhou Yang), Fundamentals of Statistical Learning (Dr. Baoxin Li), Safe Autonomy for Cyber-Physical Systems (by Dr. Georgios Fainekos), Mobile COmputing (by Dr. Ayan Bannerjee), Bio-inspired AI and Optimization (by Dr. Ted Pavlic)

Indian Institute of Information Technology Vadodara

Bachelors of Technology in Computer Science and Engineering  •  8.08/10 CGPA  •  August 2015 – July 2019