Literature review

By -

Web Scraping for Automated Water Quality
Monitoring System: A case study of PDAM
Surabaya
Rizqi Putri Nourma Budiarti , Nanang Widyatmoko , Mochamad Hariadi and Mauridhi Hery Purnomo
(Graduate Student of Electrical Engineering, Institute of Technology Sepuluh November, Surabaya, Indonesia, Department of Maintenance, PDAM Surabaya, Surabaya, Indonesia)



In recent years, many researchers have developed about online water quality monitoring capability which can provide an early warning of water pollution events. Ensuring an appropriate and timely response is the best way in detecting contamination of drinking water supplies in real time. 
In this research, the application is built by using Python programming and this system is divided into two stages.
1.     First stage:                Collecting data using web scraping tools.
2.     Second stage:          Store data in to a database system. Here the information from          web scraping is managed into columns and rows by using PostgreSQL. 
                                            
                                                 
IMPLEMENTATION
Data in a web-based application are data intensive. So, when it comes to appoint of urgent of viewing data and to get data, the operator must manually retrieve its data by using USB memory. Here they have utilized web information from the sensor output that connected in the Local Area Network .
They have used URL opens to access the sensor from server and parses the information from web pages. In here, the urllib.url open command has been made to make the browser pages navigate to the URL provided. These actions are identical to the human user like typing URL in the address bar of the browser.
By utilized looping function they have scrapped each element to an array of data. After that, they have used insert data query to store data into the database.
CONCLUSION
Their experiment shows that the accuracy of data that has been scraped from sensor are around 99%, and error data rate are less than 1%. In summary, the experimental results indicate that the correlation between data size with delay is relatively low. It means that the delay is not dependent on size, it means also that this application has been able to work almost in real-time. In the future, we have to prepare the storage facilities because the data size gradually increases. Because of that, the integration of the automated water monitoring system with Big Data visualization which can handle wide dataset is one of the future plans to be addressed. 

Comments

Post a Comment

Popular posts from this blog

KNN Algorithm

By -
Image
When do we use KNN algorithm? KNN can be used for both classification and regression predictive problems. However, it is more widely used in classification problems in the industry. To evaluate any technique we generally look at 3 important aspects: 1. Ease to interpret output 2. Calculation time 3. Predictive Power Let us take a few examples to  place KNN in the scale : KNN algorithm fairs across all parameters of considerations. It is commonly used for its easy of interpretation and low calculation time. How does the KNN algorithm work? Let’s take a simple case to understand this algorithm. Following is a spread of red circles (RC) and green squares (GS) : You intend to find out the class of the blue star (BS) . BS can either be RC or GS and nothing else. The “K” is KNN algorithm is the nearest neighbors we wish to take vote from. Let’s say K = 3. Hence, we will now make a circle with BS as center just as big as to enclose only three data points on the p...
Read more

Use amCharts to visualize Google Analytics data

By -
Image
Use amCharts to visualize Google Analytics data We can use amCharts to visualize google analytical data.Before we can extract data from your Analytics account, you will need to set up an application and API access to it and further more steps are available. An example : link : https://www.amcharts.com/kbase/use-amcharts-to-visualize-google-analytics-data/
Read more

What are the steps used in Machine Learning?

By -
What are the steps used in Machine Learning? There are 5 basic steps used to perform a machine learning task: Collecting data : Be it the raw data from excel, access, text files etc., this step (gathering past data) forms the foundation of the future learning. The better the variety, density and volume of relevant data, better the learning prospects for the machine becomes. Preparing the data : Any analytical process thrives on the quality of the data used. One needs to spend time determining the quality of data and then taking steps for fixing issues such as missing data and treatment of outliers. Exploratory analysis is perhaps one method to study the nuances of the data in details thereby burgeoning the nutritional content of the data. Training a model : This step involves choosing the appropriate algorithm and representation of data in the form of the model. The cleaned data is split into two parts – train and test (proportion depending on the p...
Read more