Machine Learning

Typical algorithms for solving reinforcement learning (RL) problems, are built on an assumption of a stationary environment (modeled as a stationary MDP), meaning the agent is learning how to act in an environment in which the action chosen in each state is not time dependent. However, one can think of many everyday life problems that occur in non-stationary environments, which change over time. Such problems were discussed in former articles...

The RL framework requires to define reward function, which generates scalar reward per action, which can be a hard task. Therefore, many RL environments are described with a ‘Sparse reward’, in which most of the agent actions would receive no reward, except for an action that would lead the agent to the final goal. A lot of RL algorithms can have difficulties getting to a good result in those kinds...

The project’s goal is to build a framework that allows adding object tracking capabilities to an existing robot. The project focused on adding those capabilities specifically to the Hexapod robot. The Hexapod robot was built in CRML lab as part of a project by previous students. In addition, the framework will allow future projects to acquire object tracking capabilities more easily for the purpose of further developments in CRML lab....

REPLAB is a reproducible and self-contained hardware stack (robot arm, camera, and workspace), that aims to drive wide participation by lowering the barrier to entry into robotics and to enable easy scaling to many robots. In this project we used REPLAB’s stack to grasp wood blocks placed on a tray and added new infrastructure for block classification by size. This new infrastructure includes both analytical, and AI based classifiers for...

Graphs are gaining increasing popularity in research and the industry due to their high representational power, which uses the interactions between different objects in the data. Different tasks can take advantage of the graph’s unique properties. Specifically, Graph Neural Networks (GNNs) are Deep Learning models where the nodes represent the neurons and propagate the data via the edges. By doing so, our forward and backward paths are both directly affected...

In most of the machine learning problems related to robotics and computer games, the algorithm learns how to perform a task with the visual representation of the state as an input, given to the RL network too choose an action for each state. A question which arises is whether the agent can get the input data in a different representation, which will yield faster running times and better average scores....

Error correcting output codes (ECOC) are commonly used to reduce multiclass classification tasks to multiple binary classification subproblems. In ECOC, classes are represented by the rows of a binary matrix, corresponding to codewords in a codebook. Commonly, given a codebook, codewords are implicitly assigned to classes arbitrarily. In this paper, we show that the traditionally-overlooked codeword-to-class assignments play a major role in the performance of ECOC. We demonstrate that assigning...

Average internet user now spends 6 hours and 40 minutes a day using the internet. While there are many apps dealing with harmful content, each user has his own personal preferences and definition for unwanted content. Project Goal • Build an application that will allow internet users to filter-out unwanted images based on their preference, in a simple and comfortable way. Transfer learning is a method where a model that...

Reinforcement learning (RL) is a popular method for solving problems involving decision tasks in which the agent has only limited  environmental  feedback.  On  top  of  RL,  curriculum learning aims to shorten the training process by training the agent on ascending difficulty environments in order to solve tasks that otherwise can’t be taught. Despite being an acceleration method, choosing the sequence of the environments is atrial and error procedure. In order...

Telemedicine or Telehealth is the distribution of health-related services and information via electronic information and telecommunication technologies. It allows long-distance patient and clinician contact, care, advice, reminders, education, intervention, monitoring, and remote admissions. In this project, we aim to build an easy to use, yet intelligent Robot assistant that will serve as an RC gateway between long-distance patients and their families, doctors, loved ones providing the patient with care, monitoring,...

Our Goal: Implement the Action Robust RL framework within the OpenAIbaselines codebase and evaluate the approach across multiple algorithms. The process: In order to deal with the research we wanted to do we started the project by learning the content relevant to the field. To this end, we first saw the RL Course lectures by David Silver. After learning the required areas in RL we read several articles including our...

Planning is branch in artificial intelligence that tries to create general solvers that can solve any problem described by the set:  . Planners try to solve the different problems by deploying a graph in which nodes represent the problem states and conducting forward search to find the optimal path from the initial state to the goal state. The number of states in a problem is exponential in the number of...

Shoppers rely on Home Depot’s product authority to find and buy the latest products and to get timely solutions to their home improvement needs. From installing a new ceiling fan to remodeling an entire kitchen, with the click of a mouse or tap of the screen, customers expect the correct results to their queries – quickly. Speed, accuracy and delivering a frictionless customer experience are essential.   In this project,...

What is Natural Language Processing? Natural language processing (NLP) is a subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data. The result is a computer capable of “understanding” the contents of documents, including the contextual nuances of the language within them. The technology can then accurately extract information and insights contained in the...

This project seeks to offer a new solution to Operating System Page Handling, the subject of page handling has been around since the dawn of computing. This project attempts to bring the technology into the modern age using Machine Learning. The Project outlines the problems facing current cache paging algorithms, and why the necessity for a modern approach is a must. There are several bottlenecks when it comes to computing,...

Our project examines a new deep exploration method in reinforcement learning. Reinforcement learning is an area of machine learning. It is about taking suitable action to maximize reward in a particular situation. There are two factors in the learning process: the agent and the environment. In reinforcement learning, the agent observes the current environment state, takes an action and gets a reward from the environment. The agent learns by observing...

Online learning is a general framework for sequential decision-making under uncertainty. On each round, the learner performs an action and observes some feedback from the environment. We apply the Online Learning framework to the context of stock exchange, what is called online portfolio selection. In this setting, at each round the learner chooses an allocation vector, named portfolio, which specifies how the money should be invested. Hence, the feedback is...

REPLAB The project’s goal is to replicate a standard, reproducible, and cheap environment for benchmarking robotic arm object grasping algorithms. The original environment was created in Berkley and reviewed in the article: “REPLAB: A Reproducible Low-Cost Arm Benchmark Platform for Robotic Learning” by Brian Yang, Jesse Zhang, Vitchyr Pong, Sergey Levine, and Dinesh Jayaraman. The standard REPLAB cell includes an arena with a wooden base, a cage comprised of metal...

Object detection is a Computer Vision task which requires to find the locations of all objects in a given image and classify the objects. Deep Neural Networks (DNNs) for object detection are in continuous development and change rapidly, with several different approaches for constructing DNNs that are suitable for the task. In this project we will first examine several approaches to DNNs for object detection and then build a system that...

Twitter is known as a platform with great influence on public opinion. Many companies operate fake twitter users (or “handles”) with the sole purpose of influencing public opinion. Such handles are tagged as “Trolls”. In order for Twitter to not block the troll handles, the operating companies of such handles try very hard to disguise them as regular handles. In this project we had built a machine learning system which...

Following Bitcoin’s popularity in recent years, use of cryptocurrency have been growing rapidly and it’s becoming more necessary to develop systems that can analyze its behavior and try to predict future prices. In this project we implemented this by using several machine learning algorithms to predict prices after an hour, a day or a week and compared their performance and results.

The goal of the project is to classify “Amazon” reviews based on “helpful/unhelpful” votes, i.e. given a review, to predict if it is “helpful” or not. The classification process in based on the number of “helpful/unhelpful” votes for each review in our dataset that contains hundreds of thousands of reviews. After preprocessing of the data, we will compare between different Machine Learning models, some of them are relatively naïve and...

One of the greatest long-lasting issues in the field of reinforcement learning is the ability to increase a model’s performance and training it on offline, partially observable data. This problem is most obvious in fields like autonomous cars and medical procedures, since access to a simulator is expensive- as it can jeopardize people’s lives, while at the same time massive amounts of data from drivers and doctors exists and is...

This project is based on the article: Watching and Acting Together: Concurrent Plan Recognition and Adaptation for Human-Robot Teams by Steven J. Levine and Brian C. Williams (2018) from MIT Computer Science and Artificial Intelligence Laboratory.   In recent years, the field of AI (Artificial Intelligence) and especially the use of robots, has been developing rapidly, among other things, for the purpose of performing work together with humans in real...

Coordinating the movement of multiple trains is a hard planning task. The computational complexity of solving such a problem is exponential in the number of trains in the general case. The project deals with solving the Flatland domain, which simulates the train coordination problem. It is done by arranging the train by priority, and planning the optimal route for each train, given the routes for the other trains. If a...

Today there is a complicated problem in the world of aviation – finding an optimal flight path while considering various constraints such as: defined routes time-limited flight zones different altitudes arrival time wind, fuel and more. Organizations are willing to invest a lot of resources in order to solve this problem. In our project we will offer implementation to solve this problem while considering those constraints and using information that...

There are many ways to solve an RL problem- we have focused on a model free algorithm. This approach has no assumption required regarding the environment. This inclusion allows us to use the same algorithm to solve different games. “A more sophisticated sampling strategy might emphasize transitions from which we can learn the most” playing atari with deep reinforcement learning Improve Q-Learning by changing the replay buffer to choose between...

In this project, we consider the problem of RL in continuous state spaces with sparse reward. In continuous state spaces tabular methods cannot be applied directly. Thus, approximation approaches, such as function approximation methods, are used. These approaches often lack in exploration compared to classical tabular RL algorithms and thus perform badly in sparse reward domains. A possible solution is by performing a discretization of the space. We explored different...

In a high-rise building, with big number of elevators, there are many ways to move passengers from their source floor to their destination (like moving different elevators to each floor, and deciding which elevator each passenger will take). Our project should learn an algorithm that chooses, for each state, the best action for all the elevators. The chosen action is the action that will probably give the highest reward for...

Mortgage applications can involve difficult decisions, for both the financial institute and for the client. The lender takes a big risk – approving mortgages for lenders who may not be able to pay them off in the future. Errors that banks have done in approving mortgages too easily have led to financial crises in the past, which emphasizes the importance of cautious decision making. The client also takes a major...

In this project we want to solve blocks-world problem. We have a table with four blocks in different colors and five spots where we can place the blocks (four of them are already taken in the given state). To this state we’ll call the initial state. Our goal is, with autonomous agent, to solve the problem. Meaning, change the blocks’ order to an order given by the user in the...

In this work, we use the LSTM version of Recurrent Neural Networks, to predict the price of Bitcoin. We describe the dataset, which is comprised of data from an API to Coinmarketcap and how its preprocessed. Further on, we show the usage of LSTM architecture with the aforementioned time series. To conclude, we outline the results of predicting Bitcoin price of the next day and display several statistical analysis.

Monitoring the number of people in a room is important for a number of reasons, especially in these days of dealing with the corona virus and the need to limit gatherings. In addition, it is necessary to maintain the privacy of the people in the room, so we would like to avoid using photographic means for the purpose of surveillance and counting. Our solution to these requirements is to use...

Mastering Blackjack is a get-into Reinforcement Learning project. Reinforcement Learning is a branch of machine learning where an agent preforms in an environment and gets rewarded for his action. The agent goal is to learn a behavior policy that will maximize his reward over time. In our project we studied the game of Blackjack, introduced RL algorithms such as Monte-Carlo Learning, Q – Learning and DQN, tested their performance with...

The Sudoku game is a number placement puzzle that gained popularity in newspapers and among the public in the last few years. In parallel, machine learning algorithms have gained massive popularity also. One way to solve Sudoku automatically is with brute-force algorithm, and trying every possible solution using a backtracking algorithm. The problem with this method is that it is very time consuming and has high complexity. In this project...

In this project we were to deliver a software infrastructure for robot motion planning. We were to implement it using the open source code library Open Motion Planning Library in python, and for Ubuntu 18 operating system. In the future the infrastructure could easily be used in more complex problems of robot Motion Planning in multi space environment with obstacles. In order to present a properly working infrastructure we were...

In this project we will examine a complementary approach to Hierarchical Reinforcement Learning, namely using context free grammars. This approach allows the agent to commit to specific temporal structures specified by a formal language over actions. By using this method, we expect to improve the performance and sample efficiency of the learning process of the agent. It will provide the ability to impose safety constraints on an agent. Furthermore, the...

Planning is an important human ability, starting at a young age. To perform this task, the brain must synchronize and combine several cognitive systems, such as controlling and processing the resulting data. In this project, we will implement an algorithm that performs efficient planning in order to find an order of the takeoffs and landings of aircraft. This is when the main aspect is finding the optimal order which will...

In this project, we simulate the use of a drone to navigate autonomously in a construction area while analyzing workers’ safety. The drone uses offline planning to plan its route over the area, takes pictures and evaluates, using a pre-trained CNN, whether there is a worker in the images and if so whether the worker has a helmet (works safely). This simulation is a proof that integration between all the...

The main goal of this project is to make comparison between different learning algorithms. The chosen algorithms were Boosting and Deep Learning. Films information dataset was downloaded from movie box office prediction Kaggle challenge. Features enrichment have been done on the mentioned dataset, most of them are simple mathematical and programmable, in order to get more features over the data which can be easy manipulated.

Future market forecasting techniques may be classified into two major categories: (1) fundamental analysis and (2) technical analysis. Fundamental analysis is based on macro-economic data such as Purchasing Power Parity (PPP), Gross Domestic Product (GDP), Balance of Payments (BOP), Purchasing Manager Index (PMI), Central Bank outcomes, etc. On the other hand, technical analysis focuses on past data and potential repeated patterns within those data. The major point here is that...

Deep Reinforcement Learning (DRL) is a thriving field in recent years as many breakthroughs were achieved in video games played by AI. Most notable is the Deep Q-Learning Network (DQN) architecture by DeepMind (acquired by Google) that solved many Atari games. In this project we will build on a recent paper “Shallow Updates for Deep Reinforcement Learning” (written by researchers from our faculty) which combines deep learning methods with “shallow”...

Reinforcement learning achieved significant results that received publication and influence, systems that were based on RL in a variety of games such as StarCraft, GO and poker(games that until recently were considered unsolvable due to it enormous number of states) reached results that surpass human performance. The most two important elements of RL are Action and State, embedding the current state of a RL environment can help reduce the dimension...

Airbnb presents listings of properties which includes detailed data and rent price, thereby it can be assumed that there is a reasonable connection between the property details and the rental price asked. This assumption can be interpreted to mathematical and statistical models, that can be learned by learning algorithms. Machine learning is the scientific study of algorithms that build a mathematical model based on sample data, known as “training data”,...

U-Net is a state-of-the-art, wildly used deep convolutional neural network for image segmentation. It consists of encoder-decoder style architecture and skip-connections between them to localize high resolution features. In this project we examined a possible improvement to the architecture by adding unsupervised learning method (auto-encoder) in U-Net architecture variant (U-Net with ResNet-34 as encoder). We used the data and evaluation system from a Kaggle competition – “Understanding Clouds from Satellite...

Project Goal: Use tweets database, stocks data and machine learning algorithms to predict future stocks movement. Our solution: We achieved our solution by 6 major steps 1. Build a database of tweets using Tweeter developer’s API – We collected ~150k Tweets from 15.12.20 to 5.3.20 using a daily run python script – The tweets were associated with 19 major companies using relevant keywords – The keywords were categorized as “Safe”...

The project deals with the forecasting of cryptocurrency on a small exchange by analyzing the same behavior on a larger exchange. The main requirement of the project was to create a platform that would fulfill the desired prediction and, in addition, enable the trading of cryptocurrencies and interface with the exchanges. In order to produce such a platform, we started with a simpler task and created a platform that interfaces...

Reinforcement Learning is a branch of machine learning where an agent preforms in an environment and gets rewarded for his action. The agent goal is to learn a behavior policy that will maximize his reward over time. Some of the research done in this field, is about expending RL to multi agent settings. The goal of this research is to solve problems where many agents preforms in the same environment,...

Image denoising is an important image processing task that experienced significant advancement, made possible by the progress in the development of convolutional neural networks and deep learning techniques. These techniques are based on learning clean images features during a training process over large datasets of clean images. In the paper Deep Image Prior it has been established that convolutional neural networks are inherently good at generating natural looking images and...

Colonoscopy has become very common in recent decades and given the fact that all referrals are examined manually by the doctors, the process of colonoscopy preparation recommendation is inefficient. Moreover, it is very difficult for an endoscopic gastroenterologist to become proficient in all available clinical guidelines and all new drugs in order to prepper the patient in the best way. In this project, an automation system as a solution is...

Meta-RL is meta-learning on reinforcement learning tasks. After trained over a distribution of tasks, the agent is able to solve a new task by developing a new RL algorithm. Train: Throw a bunch of problems with the same core structure to the model. Test: the meta-RL agent will be able to rapidly identify the key parameters of any new problem, eventually achieving optimal performance on this new problem

In this project, we propose an advanced approach in off-policy learning, where we suggest better and more sophisticated handling of the data the agent acquires to improve its performance. We apply those methods directly to Google’s novel framework- Dopamine, which trains agents employing a variety of methods to play Atari 2600 games as in Mnih et al. (2013). Since the agent acquires data samples sequentially, they are usually highly correlated....

PROJECT’S GOAL Using Reinforcement Learning to perform social laws learning in multi agents environment. These laws will enable the autonomous cars to cross intersections without accidents. UNITY AND ML-AGENTS We used Unity to model our environment and to rum simulations. In addition, we used ml-agents tool which trains intelligent agents with Reinforcement Learning via a simple method API. The ml-agnets connects the simulation environment to the learning algorithm

The goal of this project is to test a new continuous reinforcement learning algorithm in several different simulated environments. The algorithm, SIMPLE, uses a neural net to simulate the reward function for the environment’s state, similarly to the DDPG algorithm. The project’s contribution comes from its unique approach to finding an optimal action in a continuous action space. Using the neural net as a model for the reward function, it...

In many technologies today, sensors are a significant part of the set of components. Our project deals with distance or location sensors. As with all other sensors, there is also an error here in them. This error – can be split into two – a fixed offset error – which can be found by a set of measurements and then calibrated by rebooting with the offset error balance. And another...

Lung cancer stage classification using healthy tissue and tumor transcriptome In cooperation with Ori Mayer, MD PhD student, Prof. Noam Shomron’s lab, Sackler Faculty of Medicine, Tel-Aviv University Gene expression profile is available and affordable due to recent advancement in sequencing technologies. Sequencing generates a vast amount of data. Doctors cannot process and derive meaningful insights manually. The goal of this project is to build a model which predicts tumor...

Accurate forecasts of wind speed and direction are crucial to Olympic sailors to plan the optimal sailing path and win a competition. Today’s forecasts are based on numeric solutions of complex mathematical models which are inaccurate. This inaccuracy might cause the sailor to plan a sub-optimal path and lose an Olympic medal. In this project we try to build a more accurate model based on machine learning methods, in order...

In this project, we demonstrate a visualization methodology and use it to analyze the learning process of A2C agents trained on the game Atari Breakout (Atari2600). We present examples for using our method to explore how A2C agents learn to carve side-tunnels, avoid states of losing a game and other useful skill.   Our method uses a modified version of the popular nonlinear dimensionality reduction algorithm t-SNE. The modification includes...

Recent advances in Reinforcement Learning have highlighted the difficulties in learning within complex high dimensional domains. We argue that one of the main reasons that current approaches do not perform well, is that the information is represented sub-optimally. A natural way to describe what we observe, is through natural language. In this paper, we implement a natural language state representation to learn and complete tasks. Our experiments suggest that natural...

Abstract In this project we’ve implemented a music-generating neural network based on Variational Autoencoder (VAE) architecture. The network was trained on a dataset with 500 MIDI jazz files, which were represented as piano rolls. Model Data Representation Each Midi file is converted to a piano-roll: a matrix representation of music with time and pitch axes. Then, the piano-roll is divided to slices with the same size. The slices are paired...

Hyperparameteroptimizationisbothapracticalissueandaninterestingtheoreticalproblemintraining of deep architectures. Despite many recent advances the most commonly used methods almost universally involve training multiple and decoupled copies of the model, in effect sampling the hyperparameter space. We show that at negligable additional computational cost, results can be improved by sampling paths instead of points in hyperparameter space. To this end we interpret hyperparameters as controlling the level of correlated noise in the training, which can be...

We propose a DQN based technique to generalize the concept of skills. In previous papers, we have seen attempts to hand-craft skills (Chen Tessler, 2016). The idea of skills is that skills reduce the effective size of the MDP by enabling the agent to plan at a higher level. More than that, every specific task will be performed in the best way. However, skills are specific for every domain. In...

Automatically generating images according to natural language descriptions is a fundamental problem in many applications such as computer-aided design, photo editing, art generation, and video games. AttnGAN is a neural architecture that combines the attention mechanism from the field of natural language processing, and the generative adversarial networks from the field of image processing. In this project, we implemented the architecture, suggested by the original researchers, trained the network on...

In this project we implemented a trading agent for cryptographic currencies, which uses the currencies price and volume history, as well as published posts and comments on Reddit forums. Its target is to change the portfolio appropriately, in order to maximize the profit. In the project we rely on the assumption that a correlation exists between the currency’s price and the textual publications about it on the different forums. To...

We address the problem of solving high dimensional MDPs using its minimization by sampling and solving a navigation problem on a graph using RMAX. Many previous attempts optimize the policy using deep models with some success but lack the theoretical guaranties. Our methods provide a framework to incorporate classical reinforcement learning algorithms which provide guaranties. Moreover, the method can be extended using deep models to solve more specific problems. The...

In our Project we created a system that can independently download movies reviews, classify them and decide for each review if it is positive or negative. The process of getting the reviews is done by an independent script (crawler) that can search in the internet and download a large amount of reviews. For classifying each review, we created a trained module based on machine learning algorithms with different kinds of...

We consider a distributed reinforcement learning framework where multiple agents interact with the environment in parallel, while sharing experience, in order to find the optimal policy. At each time step, only a subset of the agents is selected to interact with the environment. We explore several mechanisms for selecting which agents to prioritize based on the reward and the TD-error, and analyze their effect on the learning process. When the...

NeuroEvolution is a field in AI which uses evolutionary algorithms and can be used to solve reinforcement learning problems. The process in those algorithms is initializing simple neural net (usually on hidden neuron only) and then performing evolution on the weights space to get the best fitness score. This project is based on a paper [1] that shows that evolution of the topology can improve the performance in control tasks...

Project’s main objective was to develop a methodology for solving planning problems involving time by using simple planning tools. During the project I studied the field of classical and temporal planning, learned the PDDL language which is used for describing planning problems and designed a method for solving temporal planning problems using classical planning solver written in python (Pyperplan).

In this project, we consider the Inverse Reinforcement Learning problem in Contextual Markov Decision Processes. Here, the reward of the environment depends on a hidden static parameter referred to as the context, i.e., each context defines an MDP. The agent does not observe the reward, but instead, is provided with expert demonstrations for different contexts. The goal of the agent is to learn a mapping from contexts to rewards that...