Strumenti Utente

Strumenti Sito


mds:ddam:ddam20202021

Distributed Data Analysis and Mining (DDAM)

Docente - Teacher: Roberto Trasarti
roberto [dot] trasarti [at] isti [dot] cnr [dot] it

Corso di Laurea - Graduate Course:
DATA SCIENCE AND BUSINESS INFORMATICS

Comunicazioni - News

Orario Lezioni - Lessons Schedule

Giorno Ora Luogo
Monday14:15 - 16:00 On-line
Wednesday14:15 - 16:00 On-Line

Prima Lezione - First Lecture: 16/09/2020

Link teams classroom: https://teams.microsoft.com/l/channel/19%3a4500e29b9644403bb4333a24473979e9%40thread.tacv2/Generale?groupId=1a3cfb99-a6dd-467c-ad9f-e745714b1f98&tenantId=c7456b31-a220-47f5-be52-473828670aa1

Scopo del Corso - Aim of the course

Italiano:

Il Data Mining sui Big data è oggi un’area di ricerca molto attiva. L'applicazione delle attuali metodologie analitiche e strumenti software su un singolo personal computer non può gestire in modo efficiente dataset di grandi dimensioni. Le piattaforme di calcolo distribuito sono una soluzione scalabile per il big data mining, attraverso la scomposizione del problema in operazioni più piccole che possono essere eseguite parallelamente su singoli processori / macchine. Il corso propone l’insegnamento di concetti base del paradigma di calcolo distribuito tramite MapReduce dal punto di vista teorico e pratico, in particolare ci si focalizzerà su Hadoop per lo sviluppo di competenze nell'uso di strumenti di calcolo ad alte prestazioni per il data engineering, l’analisi di dati e l’utilizzo di tecniche di data mining. Gli studenti impareranno come i classici algoritmi di data mining possono essere applicati sui Big Data usando Hadoop (Spark). Set di dati reali (e open source) verranno utilizzati per presentare esempi e per consentire agli studenti di costruire i propri progetti. Una metà delle lezioni consisterà in esercitazioni (laboratorio) e una metà delle lezioni sarà teorica.

English:

Mining with big data or big data mining has become an active research area. Running current analytical methodologies and software tools on a single personal computer cannot efficiently deal with very large datasets. Distributed computing platforms are a scalable solution for big data mining, obtained by dividing a large problem into smaller ones that are concurrently solved by many single processor/machine. This course aims at teaching the basic theoretical concepts behind the MapReduce distributed computing paradigm, and Hadoop in particular, and at building expertise in the practical usage of high-performance computing tools for data engineering, analysis and mining. In particular, the students will learn how classical data mining algorithms can be applied to Big Data using Hadoop (Spark). Real (and open source) datasets will be used to present examples and to let the students build their own projects. Half of the lessons will consist of practice (Lab), and half of the lectures.

Syllabus:

  • Motivations: What is and Why Distributed Data Mining is needed in a Big Data Scenario
  • Recall parallel and distributed computing notions
  • Amdahl's law, differences between shared and distributed memory architectures
  • Introduction to Hadoop
  • Hadoop Ecosystem
  • Interacting with HDFS
  • Hadoop Combiners
  • Basic Spark and RDD
  • Map-Reduce Programming Patterns
  • Recall Python programming
  • Data Analysis with Spark
  • Data Mining and Machine Learning with Spark
  • Example on how to prepare a project

Registro delle Lezioni - Lessons log (in english)

(Slides and materials: in the teams classroom repository)

  • Wednesday 16/09 - Parallel Computation basic concepts, Map Reduce I (lecture 1)
  • Wednesday 23/09 - Map Reduce II, Access to Servers (lecture 2)
  • Wednesday 30/09 - Map Reduce III, Spark I (lecture 3)
  • Monday 5/10 - Spark II (lecture 4)
  • Wednesday 7/10 - Exercises (lecture 5)
  • Monday 12/10 - Spark SQL (lecture 6)
  • Wednesday 14/10 - MLlib I (lecture 7)
  • Monday 19/10 - MLlib II + Exercises (lecture 8)
  • Wednesday 21/10 - Exercises/Example Projects (lecture 9)
  • Break!
  • Wednesday 4/11 - Mlib + Spark Streaming (lecture 10)
  • Monday 9/11 - Project idea presentation (Students presentation 10 minutes each)
  • Wednesday 11/11 - Open Lab
  • Monday 16/11 - Open Lab
  • Wednesday 18/11 - Open Lab
  • Monday 23/11 - Open Lab
  • Wednesday 25/11 - Open Lab
  • Monday 30/11 - Open Lab
  • Wednesday 2/12 - Open Lab
  • Monday 7/12 - Project Alignment (Students presentation 10 minutes each)

Metodo di valutazione - Examination process

Students groups made of 3 o 4 students (max) develop a project (report + python notebooks);
https://docs.google.com/spreadsheets/d/1bMRdG6SFE4Xn492Rvpm_AfisLLyFutecOplCQo_9-SE/edit#gid=0

Laboratory

Access to a remote server with pre-installed software will be provided.

Edizioni Precedenti

Pagina dedicata Anno accademico di riferimento
Distributed Data Analysis and Mining 2019-2020
mds/ddam/ddam20202021.txt · Ultima modifica: 05/09/2021 alle 14:06 (3 anni fa) da Roberto Trasarti