A mathematical programming approach for resource allocation of data analysis workflows on heterogeneous clusters

Scientific communities are motivated to schedule their large-scale data analysis workflows in heterogeneous cluster environments because of privacy and financial issues. In such environments containing considerably diverse resources, efficient resource allocation approaches are essential for reaching high performance. Accordingly, this research addresses the scheduling problem of workflows with bag-of-task form to minimize total runtime (makespan). To this aim, we develop a mixed-integer linear programming model (MILP). The proposed model contains binary decision variables determining which tasks should be assigned to which nodes. Also, it contains linear constraints to fulfill the tasks requirements such as memory and scheduling policy. Comparative results show that our approach outperforms related approaches in most cases. As part of the post-optimality analysis, some secondary preferences are imposed on the proposed model to obtain the most preferred optimal solution. We analyze the relaxation of the makespan in the hope of significantly reducing the number of consumed nodes

Pengembangan Jadwal Shift Staf Editor Video pada Stasiun Televisi Nasional Trans7 berbasis Android menggunakan Algoritma Ant Colony dengan Firebase

Perusahaan bisnis bidang broadcasting atau tepatnya stasiun televisi memiliki strategi untuk menayangkan tayangan acara yang paling ditunggu oleh penonton setianya, yaitu berupa tayangan acara yang up to date dan cepat dalam penyiarannya. Oleh sebab itu, diperlukan staf pada bagian pasca produksi untuk bekerja secara cepat dan sesuai dengan prosedur tayangan. Untuk mempersiapkan tayangan terbaru secara cepat, diperlukan sistem penjadwalan staf editor video yang jumlahnya besar secara cepat, tepat, dan dapat dilihat dengan instan oleh para editor video sehingga memudahkan editing video setiap harinya. Algoritma yang cocok untuk menghasilkan jadwal secara cepat dengan jumlah yang besar dengan tidak ada data yang bentrok adalah ant colony atau algoritma koloni semut. Algoritma ant colony ini mengacu pada cara hidup semut yang berkelompok dalam mencari makanan sehingga dapat kembali lagi ke tempat semula dengan jalur yang sama dan cepat. Data masukan yang digunakan dalam penelitian ini adalah nama editor dan ruangan, serta luaran berupa jadwal per shift untuk setiap ruangan untuk suatu periode tertentu. Penelitian ini menggunakan basis data MySQL dan firebase. Jadwal editor yang telah diolah pada aplikasi back-end kemudian diubah ke dalam bentuk aplikasi android, dengan demikian jadwal tersebut dapat dilihat oleh seluruh staf editor video melalui smartphone masing-masing. Pengujian dilakukan terhadap hasil perhitungan algoritma, fungsional sistem, integrasi, dan penerimaan yang dikembalikan pada sistem dengan mencari kesalahan pada interface perangkat lunak, dan pengujian penggunaan langsung kepada pengguna. Hasil pengujian menunjukkan bahwa algoritma ant colony dapat digunakan untuk menyusun jadwal editor video dengan cepat dan tepat, semua fitur berjalan dengan baik, dan tingkat kepuasan pengguna cukup tinggi. AbstractBroadcasting or television business companies have a strategy to broadcast programs up to date and quickly. Therefore, staff in the post-production section are required to work quickly and in accordance with the show procedures. To prepare the latest shows quickly, a video editor staff scheduling system is needed. That scheduling system should be fast, precise, and can be seen instantly by video editors so that the editing processes can be done easily every day. The suitable algorithm to generate a schedule quickly in large numbers with no data clashing is ant colony.Ant colony algorithm refers to the way of ants when looking for food in a group, then they can return again to their original place with the same path and fast. The input data used in this study is the name of the editor and the room, and output in the form of a schedule per shift for each room for a certain period. This research uses MySQL and firebase databases. The editor's schedule that has been processed in the back-end application is then converted into an android application, therefore the schedule can be seen by all video editor staff via their own smartphones. Tests are carried out on the results of algorithm calculations, functional systems, integration, and feedback to the system by looking for errors on the software interface, and direct use testing to users. The test results show that the Ant Colony algorithm can be used to compile a schedule of video editors quickly and precisely, all features run well, and the level of user satisfaction is quite high

Exploring Multi-Reader Buffers and Channel Placement during Dataflow Network Mapping to Heterogeneous Many-core Systems

This paper presents an approach for reducing the memory requirements of dataflow applications, while minimizing the execution period when deployed on a many-core target. Often, straightforward implementations of dataflow applications suffer from data duplication if identical data has to be processed by multiple actors. In fact, multi-cast actors can produce huge memory overheads when storing and communicating copies of the same data. As a remedy, so-called Multi-Reader Buffers (MRBs) can be utilized to forward identical data to multiple actors in a FIFO manner while storing each data item only once. However, MRBs may increase the achievable period due to communication contention when accessing the shared data. A novel multi-objective design space exploration approach is proposed that selectively replaces multi-cast actors with MRBs and explores actor and FIFO channel mappings to find trade-offs between the objectives of period, memory footprint, and core cost. Our approach considers (i) memory-size constraints, (ii) hierarchical memories to implement the buffers, (iii) supports heterogeneous many-core platforms, and (iv) optimizes the buffer placement and overall scheduling to minimize the execution period by proposing a novel combined actor and communications scheduling heuristic for period minimization called CAPS-HMS. Our results show that the explored Pareto fronts improve a hypervolume indicator over a reference approach by up to 66 % for small to mid-size applications and 90 % for large applications. Moreover, selectively replacing multi-cast actors with corresponding MRBs proves to be always superior to never or always replacing them. Finally, it is shown that the quality of the explored Pareto fronts does not degrade when replacing the efficient scheduling heuristic CAPS-HMS by an exact integer linear programming (ILP) solver.Comment: 20 page