Our Flagship product Freespace is installed in over 40 cities around the world and serves over 30000 people every day

Freespace

freespace

Authors:
(1) Renan F. F. da Silva, Institute of Computing, University of Campinas;
(2) Yulle G. F. Borges, Institute of Computing, University of Campinas;
(3) Rafael C. S. Schouery, Institute of Computing, University of Campinas. 6 Variable Neighborhood Search 6.1 Objective function Our objective function f has two levels: the primary objective that seeks to minimize the number of used bins and the secondary objective that seeks the lowest lexicographic order in the residual vector. The residual vector consists of the residual capacity of used bins in non-decreasing order. 6.2 Neighborhoods The VNS performs local searches in neighborhoods, as will be explained below. One relevant factor is the heuristic used in the local search. In our VNS, we use the Best Improvement heuristic, which analyzes all neighbors of a solution and changes to the best neighbor (in terms of the objective function). Our VNS has neighborhoods with the moves: Move-Item, Swap-Items, Swap-and-Move, and MoveTwo-to-One. The number of items changed in one move is constant for these neighborhoods, which just change a constant number of bins and elements in the residual vector (of the second objective function). As we can compare two neighborhoods using their variations in the objective function, it can be done in O(1). Next, we present our neighborhoods, showing optimizations to obtain fast algorithms even for large instances. 6.2.1 Move-Item Neighborhood For the CBPP, one possible algorithm for this neighborhood is to adapt the algorithm used by Fleszar and Hindi [19] for the BPP, that have time complexity O(n2). In this, for each item i, we iterate in the bins given in non-ascending order of residual capacity. Then, we check if the move (i, Br) is valid, where Br is the current bin, and evaluate it if so. Besides, we stop the search when we find the first bin Br such that wi > res(r), since the ordering guarantee that the bins forward also do not have space for the item i. However, it is not necessary to evaluate all moves to find the best one. Applying the Auxiliary Algorithm using the current solution B, it returns the best move for each item i ∈ I, so the best one is the best move between them. It gives us an implementation of the Move-Item Neighborhood with time complexity Θ(n). 6.2.2 Swap-Items Neighborhood Fleszar and Hindi [19] propose for the BPP a Swap-Items Neighborhood with complexity O(n+M), where M is the number of valid moves, which can be very fast in some solutions with little free space. We present the pseudocode of this neighborhood in the Algorithm 4. Although the time complexity of this algorithm is O(n log n + M), in our tests, it performed better than Fleszar and Hindi’s algorithm, especially in large instances. Algorithm 5 presents the pseudocode of this algorithm. 6.2.3 Move-Two-to-One Neighborhood 6.2.4 Swap-and-Move Neighborhood The Swap-and-Move Neighborhood selects three items i, j, k from different bins, then swaps i and j, and moves k to B(i) (its bin before the move). This neighborhood has complexity O(n3) when implemented in the completed way, which offers a low cost-benefit, so we use a restricted version from this one. Instead of testing all the possibilities of k, we test only the heaviest item that can be moved to B(i). The first candidate for k is the heaviest item that fits in B(i), and we can use a binary search to find it. 6.3 Shake Algorithm The Shake Algorithm is a function that makes random moves, seeking to escape from local optimal. In particular, our Shake chooses to execute one of two subroutines with equal probability. In the first subroutine, we performed 20 moves with the constraint that each bin belongs to a maximum of one move. The possible moves are to move one item or swap two items. In the second subroutine, we select two bins and then repack their items in a random order using the Best Fit heuristic. Note that we use a very aggressive Shake since its moves can make the solution much worse. Our Shake is based on the one proposed by Fleszar and Hindi [19] for the BPP, which is equal to the first subroutine without the constraint that bins must be distinct in different moves. The second routine was added because the perturbation generated by the first one can be small in instances with little free space. 6.4 Algorithm 6.5 Neighborhood Ordering This paper is available on arxiv under CC 4.0 license. Authors: (1) Renan F. F. da Silva, Institute of Computing, University of Campinas; (2) Yulle G. F. Borges, Institute of Computing, University of Campinas; (3) Rafael C. S. Schouery, Institute of Computing, University of Campinas. Authors: (1) Renan F. F. da Silva, Institute of Computing, University of Campinas; (2) Yulle G. F. Borges, Institute of Computing, University of Campinas; (3) Rafael C. S. Schouery, Institute of Computing, University of Campinas. 6 Variable Neighborhood Search 6.1 Objective function Our objective function f has two levels: the primary objective that seeks to minimize the number of used bins and the secondary objective that seeks the lowest lexicographic order in the residual vector. The residual vector consists of the residual capacity of used bins in non-decreasing order. 6.2 Neighborhoods The VNS performs local searches in neighborhoods, as will be explained below. One relevant factor is the heuristic used in the local search. In our VNS, we use the Best Improvement heuristic, which analyzes all neighbors of a solution and changes to the best neighbor (in terms of the objective function). Our VNS has neighborhoods with the moves: Move-Item, Swap-Items, Swap-and-Move, and MoveTwo-to-One. The number of items changed in one move is constant for these neighborhoods, which just change a constant number of bins and elements in the residual vector (of the second objective function). As we can compare two neighborhoods using their variations in the objective function, it can be done in O(1). Next, we present our neighborhoods, showing optimizations to obtain fast algorithms even for large instances. 6.2.1 Move-Item Neighborhood For the CBPP, one possible algorithm for this neighborhood is to adapt the algorithm used by Fleszar and Hindi [19] for the BPP, that have time complexity O(n2). In this, for each item i, we iterate in the bins given in non-ascending order of residual capacity. Then, we check if the move (i, Br) is valid, where Br is the current bin, and evaluate it if so. Besides, we stop the search when we find the first bin Br such that wi > res(r), since the ordering guarantee that the bins forward also do not have space for the item i. However, it is not necessary to evaluate all moves to find the best one. Applying the Auxiliary Algorithm using the current solution B, it returns the best move for each item i ∈ I, so the best one is the best move between them. It gives us an implementation of the Move-Item Neighborhood with time complexity Θ(n). 6.2.2 Swap-Items Neighborhood Fleszar and Hindi [19] propose for the BPP a Swap-Items Neighborhood with complexity O(n+M), where M is the number of valid moves, which can be very fast in some solutions with little free space. We present the pseudocode of this neighborhood in the Algorithm 4. Although the time complexity of this algorithm is O(n log n + M), in our tests, it performed better than Fleszar and Hindi’s algorithm, especially in large instances. Algorithm 5 presents the pseudocode of this algorithm. 6.2.3 Move-Two-to-One Neighborhood 6.2.4 Swap-and-Move Neighborhood The Swap-and-Move Neighborhood selects three items i, j, k from different bins, then swaps i and j, and moves k to B(i) (its bin before the move). This neighborhood has complexity O(n3) when implemented in the completed way, which offers a low cost-benefit, so we use a restricted version from this one. Instead of testing all the possibilities of k, we test only the heaviest item that can be moved to B(i). The first candidate for k is the heaviest item that fits in B(i), and we can use a binary search to find it. 6.3 Shake Algorithm The Shake Algorithm is a function that makes random moves, seeking to escape from local optimal. In particular, our Shake chooses to execute one of two subroutines with equal probability. In the first subroutine, we performed 20 moves with the constraint that each bin belongs to a maximum of one move. The possible moves are to move one item or swap two items. In the second subroutine, we select two bins and then repack their items in a random order using the Best Fit heuristic. Note that we use a very aggressive Shake since its moves can make the solution much worse. Our Shake is based on the one proposed by Fleszar and Hindi [19] for the BPP, which is equal to the first subroutine without the constraint that bins must be distinct in different moves. The second routine was added because the perturbation generated by the first one can be small in instances with little free space. 6.4 Algorithm 6.5 Neighborhood Ordering This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Understanding Variable Neighborhood Search

This paper is available on arxiv under CC 4.0 license. Authors: (1) Xiaoyu Ai, School of Electrical Engineering & Telecommunications, University of New South Wales, Sydney, NSW 2052, Australia; (2) Robert Malaney, School of Electrical Engineering & Telecommunications, University of New South Wales, Sydney, NSW 2052, Australia. Table of Links Abstract & Introduction System Overview Overcoming the limitation of key reconciliation Final Key Rate Experimental Results Discussion Conclusion & References Appendix VII. CONCLUSION In this work, we have carried out a full-blown analysis and experimental implementation of a Slice Reconciliation scheme applied to a specific CV QKD protocol (with post-selection) under simulated channel conditions anticipated for satelliteto-Earth channels. We have provided the optimal solution for the classical reconciliation process for this CV-QKD protocol in the context of massive parallelisation under the finite key regime. More specifically, we have identified the optimal block length when a large-code block is to be subdivided so as to improve the final secure key rate in bits per second. Although our results were based on a specific CV-QKD protocol and a specific GPU architecture, the type of analysis we have introduced here will apply in general terms a large suite of CV-QKD protocols run over any form of architecture that offers massive parallelisation. Our results, therefore, pave the way to optimal reconciliation system design for a wide range of practical CV-QKD systems that operate in the finite key regime. As the demand on the finite key size grows (better security thresholds), and technology advances lead to larger quantum signalling rates, the importance of optimised multithreaded CV-QKD reconciliation will grow. REFERENCES [1] N. Hosseinidehaj, Z. Babar, R. Malaney, S. X. Ng, and L. Hanzo, “Satellite-Based Continuous-Variable Quantum Communications: Stateof-the-Art and a Predictive Outlook,” IEEE Communications Surveys & Tutorials, vol. 21, no. 1, 881–919, 2018. [2] C. H. Bennett and G. Brassard, “Quantum Cryptography: Public Key Distribution and Coin Tossing,” in Proceedings of IEEE International Conference on Computers, Systems and Signal Processing,, vol. 175, 8, 1984. [3] H. Weinfurter, “Quantum Communication Experiments with Discrete Variables,” Quantum Information: From Foundations to Quantum Technology Applications, 369–381, 2016. [4] L. Gy ¨ongy ¨osi, L. Bacsardi, and S. Imre, “A Survey on Quantum Key Distribution,” Infocommunications Journal, vol. 11, no. 2, 14–21, 2019. [5] S. Pirandola, U. L. Andersen, L. Banchi, M. Berta, D. Bunandar, R. Colbeck, D. Englund, T. Gehring, C. Lupo, C. Ottaviani, et al., “Advances in Quantum Cryptography,” Advances in Optics and Photonics, vol. 12, no. 4, 1012–1236, 2020. [6] B. Korzh, C. C. W. Lim, R. Houlmann, N. Gisin, M. J. Li, D. Nolan, B. Sanguinetti, R. Thew, and H. Zbinden, “Provably Secure and Practical Quantum Key Distribution over 307 km of Optical Fibre,” Nature Photonics, vol. 9, no. 3, 163–168, 2015. [7] T. A. Eriksson, T. Hirano, B. J. Puttnam, G. Rademacher, R. S. Lu´ıs, M. Fujiwara, R. Namiki, Y. Awaji, M. Takeoka, N. Wada, et al., “Wavelength Division Multiplexing of Continuous Variable Quantum Key Distribution and 18.3 Tbit/s Data Channels,” Communications Physics, vol. 2, no. 1, 1–8, 2019. [8] S.-Y. Shen, M.-W. Dai, X.-T. Zheng, Q.-Y. Sun, G.-C. Guo, and Z.- F. Han, “Free-Space Continuous-Variable Quantum Key Distribution of Unidimensional Gaussian Modulation Using Polarized Coherent States in an Urban Environment,” Physical Review A, vol. 100, 012325, 2019. [9] L. Gyongyosi and S. Imre, “Secret Key Rates of Free-Space Optical Continuous-Variable Quantum Key Distribution,” International Journal of Communication Systems, vol. 32, no. 18, e4152, 2019. [10] P. Jouguet, S. Kunz-Jacques, and A. Leverrier, “Long-Distance Continuous-Variable Quantum Key Distribution with a Gaussian Modulation,” Physical Review A, vol. 84, no. 6, 062317, 2011. [11] P. Jouguet, S. Kunz-Jacques, T. Debuisschert, S. Fossier, E. Diamanti, R. All´eaume, R. Tualle-Brouri, P. Grangier, A. Leverrier, P. Pache, et al., “Field Test of Classical Symmetric Encryption with Continuous Variables Quantum Key Distribution,” Optics Express, vol. 20, no. 13, 14030–14041, 2012. [12] Q. Liao, Y. Guo, D. Huang, P. Huang, and G. Zeng, “Long-Distance Continuous-Variable Quantum Key Distribution Using Non-Gaussian State-Discrimination Detection,” New Journal of Physics, vol. 20, no. 2, 023015, 2018. [13] D. Guo, C. He, T. Guo, Z. Xue, Q. Feng, and J. Mu, “Comprehensive High-Speed Reconciliation for Continuous-Variable Quantum Key Distribution,” Quantum Information Processing, vol. 19, no. 9, 1–19, 2020. [14] Y. Zhang, Z. Chen, S. Pirandola, X. Wang, C. Zhou, B. Chu, Y. Zhao, B. Xu, S. Yu, and H. Guo, “Long-Distance Continuous-Variable Quantum Key Distribution over 202.81 km of Fiber,” Physical Review Letters, vol. 125, no. 1, 010502, 2020. [15] A. Leverrier, F. Grosshans, and P. Grangier, “Finite-Size Analysis of a Continuous-Variable Quantum Key Distribution,” Physical Review A, vol. 81, no. 6, 062343, 2010. [16] F. Furrer, T. Franz, M. Berta, A. Leverrier, V. B. Scholz, M. Tomamichel, and R. F. Werner, “Continuous Variable Quantum Key Distribution: Finite-Key Analysis of Composable Security against Coherent Attacks,” Physical Review Letters, vol. 109, no. 10, 100502, 2012. [17] S. Kish, E. Villase˜nor, R. Malaney, K. Mudge, and K. Grant, “Feasibility Assessment for Practical Continuous Variable Quantum Key Distribution over the Satellite-to-Earth Channel,” Quantum Engineering, vol. 2, no. 3, e50, 2020. [18] M. Tomamichel, C. C. W. Lim, N. Gisin, and R. Renner, “Tight FiniteKey Analysis for Quantum Cryptography,” Nature Communications, vol. 3, no. 1, 1–6, 2012. [19] Z. Yuan, A. Plews, R. Takahashi, K. Doi, W. Tam, A. Sharpe, A. Dixon, E. Lavelle, J. Dynes, A. Murakami, et al., “10-Mb/s Quantum Key Distribution,” Journal of Lightwave Technology, vol. 36, no. 16, 3427– 3433, 2018. [20] S.-K. Liao, W.-Q. Cai, W.-Y. Liu, L. Zhang, Y. Li, J.-G. Ren, J. Yin, Q. Shen, Y. Cao, Z.-P. Li, et al., “Satellite-to-Ground Quantum Key Distribution,” Nature, vol. 549, no. 7670, 43–47, 2017. [21] D. Lin, D. Huang, P. Huang, J. Peng, and G. Zeng, “High Performance Reconciliation for Continuous-Variable Quantum Key Distribution with LDPC Code,” International Journal of Quantum Information, vol. 13, no. 02, 1550010, 2015. [22] X. Wang, Y. Zhang, S. Yu, B. Xu, Z. Li, and H. Guo, “Efficient Rate-Adaptive Reconciliation for Continuous-Variable Quantum Key Distribution,” Quantum Information & Computation, vol. 17, no. 13– 14, 1123–1134, 2017. [23] C. Zhou, X. Wang, Y. Zhang, Z. Zhang, S. Yu, and H. Guo, “ContinuousVariable Quantum Key Distribution with Rateless Reconciliation Protocol,” Physical Review Applied, vol. 12, no. 5, 054013, 2019. [24] M. Milicevic, C. Feng, L. M. Zhang, and P. G. Gulak, “Quasi-Cyclic Multi-Edge LDPC Codes for Long-Distance Quantum Cryptography,” NPJ Quantum Information, vol. 4, no. 1, 1–9, 2018. [25] S.-S. Yang, Z.-G. Lu, and Y.-M. Li, “High-Speed Post-Processing in Continuous-Variable Quantum Key Distribution Based on FPGA Implementation,” Journal of Lightwave Technology, vol. 38, no. 15, 3935–3941, 2020. [26] H. Li and Y. Pang, “FPGA-Accelerated Quantum Computing Emulation and Quantum Key Distillation,” IEEE Micro, vol. 41, no. 4, 49–57, 2021. [27] C. Weedbrook, A. M. Lance, W. P. Bowen, T. Symul, T. C. Ralph, and P. K. Lam, “Quantum Cryptography without Switching,” Physical Review Letters, vol. 93, no. 17, 170504, 2004. [28] N. Hosseinidehaj, A. M. Lance, T. Symul, N. Walk, and T. C. Ralph, “Finite-Size Effects in Continuous-Variable Quantum Key Distribution with Gaussian Postselection,” Physical Review A, vol. 101, no. 5, 052335, 2020. [29] D. Dequal, L. T. Vidarte, V. R. Rodriguez, G. Vallone, P. Villoresi, A. Leverrier, and E. Diamanti, “Feasibility of Satellite-to-Ground Continuous-Variable Quantum Key Distribution,” NPJ Quantum Information, vol. 7, no. 1, 1–10, 2020. [30] M. Bloch, A. Thangaraj, and S. W. McLaughlin, “Efficient Reconciliation of Correlated Continuous Random Variables Using LDPC Codes,” arXiv preprint cs/0509041, 2005. [31] P. Jouguet, D. Elkouss, and S. Kunz-Jacques, “High-Bit-Rate Continuous-Variable Quantum Key Distribution,” Physical Review A, vol. 90, no. 4, 042329, 2014. [32] H. Mani, T. Gehring, P. Grabenweger, B. Omer, C. Pacher, and ¨ U. L. Andersen, “Multiedge-Type Low-Density Parity-Check Codes for Continuous-Variable Quantum Key Distribution,” Physical Review A, vol. 103, no. 6, 062419, 2021. [33] H. Imai and S. Hirakawa, “A New Multilevel Coding Method Using Error-Correcting Codes,” IEEE Transactions on Information Theory, vol. 23, no. 3, 371–377, 1977. [34] U. Wachsmann, R. F. Fischer, and J. B. Huber, “Multilevel codes: Theoretical concepts and practical design rules,” IEEE Transactions on Information Theory, vol. 45, no. 5, 1361–1391, 1999. [35] R. D. Wesel, X. Liu, J. M. Cioffi, and C. Komninakis, “Constellation Labeling for Linear Encoders,” IEEE Transactions on Information Theory, vol. 47, no. 6, 2417–2431, 2001. [36] M. Bloch, A. Thangaraj, S. W. McLaughlin, and J.-M. Merolla, “LDPCBased Gaussian Key Reconciliation,” in IEEE Information Theory Workshop-ITW’06 Punta del Este, 116–120, IEEE, 2006. [37] A. Leverrier, R. All´eaume, J. Boutros, G. Z´emor, and P. Grangier, “Multidimensional Reconciliation for a Continuous-Variable Quantum Key Distribution,” Physical Review A, vol. 77, no. 4, 042325, 2008. [38] J. Fiur´aˇsek and N. J. Cerf, “Gaussian Postselection and Virtual Noiseless Amplification in Continuous-Variable Quantum Key Distribution,” Physical Review A, vol. 86, no. 6, 060302, 2012. [39] F. Darema, “The SPMD Model: Past, Present and Future,” in European Parallel Virtual Machine/Message Passing Interface Users’ Group Meeting, 1–1, Springer, 2001. [40] M. Pharr and W. R. Mark, “A SPMD Compiler for High-Performance CPU Programming,” in Innovative Parallel Computing (InPar), 1–13, IEEE, 2012. [41] J. Lodewyck, M. Bloch, R. Garc´ıa-Patr ´on, S. Fossier, E. Karpov, E. Diamanti, T. Debuisschert, N. J. Cerf, R. Tualle-Brouri, S. W. McLaughlin, et al., “Quantum Key Distribution over 25 km with an AllFiber Continuous-Variable System,” Physical Review A, vol. 76, no. 4, 042305, 2007. [42] F. Laudenbach, C. Pacher, C.-H. F. Fung, A. Poppe, M. Peev, B. Schrenk, M. Hentschel, P. Walther, and H. H ¨ubel, “Continuous-Variable Quantum Key Distribution with Gaussian Modulation—The Theory of Practical Implementations,” Advanced Quantum Technologies, vol. 1, no. 1, 1800011, 2018. [43] Y. Polyanskiy, H. V. Poor, and S. Verd ´u, “Channel Coding Rate in the Finite Blocklength Regime,” IEEE Transactions on Information Theory, vol. 56, no. 5, 2307–2359, 2010. [44] D. Slepian and J. Wolf, “Noiseless Coding of Correlated Information Sources,” IEEE Transactions on Information Theory, vol. 19, no. 4, 471–480, 1973. [45] T. Richardson and R. Urbanke, Modern Coding Theory. Cambridge University Press, 2008. [46] X. Ai, R. Malaney, and S. X. Ng, “A Reconciliation Strategy for RealTime Satellite-Based QKD,” IEEE Communications Letters, vol. 24, no. 5, 1062–1066, 2020. [47] S.-Y. Chung, T. J. Richardson, and R. L. Urbanke, “Analysis of SumProduct Decoding of Low-Density Parity-Check Codes Using a Gaussian Approximation,” IEEE Transactions on Information Theory, vol. 47, no. 2, 657–670, 2001. [48] V. A. Chandrasetty and S. M. Aziz, “FPGA Implementation of an LDPC Decoder Using a Reduced Complexity Message Passing Algorithm,” Journal of Networks, vol. 6, no. 1, 36, 2011. [49] A. Leverrier, “Composable Security Proof for Continuous-Variable Quantum Key Distribution with Coherent States,” Physical Review Letters, vol. 114, no. 7, 070501, 2015. [50] C. Lupo, C. Ottaviani, P. Papanastasiou, and S. Pirandola, “ContinuousVariable Measurement-Device-Independent Quantum Key Distribution: Composable Security against Coherent Attacks,” Physical Review A, vol. 97, no. 5, 052327, 2018. [51] N. Hosseinidehaj, N. Walk, and T. C. Ralph, “Optimal Realistic Attacks in Continuous-Variable Quantum Key Distribution,” Physical Review A, vol. 99, no. 5, 052336, 2019. [52] S. Pirandola, “Limits and Security of Free-Space Quantum Communications,” Physical Review Research, vol. 3, no. 1, 013279, 2021. [53] S. Pirandola, “Satellite Quantum Communications: Fundamental Bounds and Practical Security,” Physical Review Research, vol. 3, no. 2, 023130, 2021. [54] J. H. Mathews and K. D. Fink, Numerical Methods Using MATLAB, vol. 4. Pearson Prentice Hall Upper Saddle River, NJ, 2004. [55] A. Leverrier, “Security of Continuous-Variable Quantum Key Distribution via a Gaussian de Finetti Reduction,” Physical Review Letters, vol. 118, no. 20, 200501, 2017. [56] E. Villase˜nor, R. Malaney, K. A. Mudge, and K. J. Grant, “Atmospheric Effects on Satellite-to-Ground Quantum Key Distribution Using Coherent States,” in GLOBECOM 2020-2020 IEEE Global Communications Conference, 1–6, IEEE, 2020. [57] J. Mateo, Efficient Information Reconciliation for Quantum Key Distribution. PhD thesis, Universidad Politecnica de Madrid, 2011. [58] T. Tian, C. R. Jones, J. D. Villasenor, and R. D. Wesel, “Selective Avoidance of Cycles in Irregular LDPC Code Construction,” IEEE Transactions on Communications, vol. 52, no. 8, 1242–1247, 2004. [59] D. V. Nguyen, S. K. Chilappagari, M. W. Marcellin, and B. Vasic, “On the Construction of Structured LDPC Codes Free of Small Trapping Sets,” IEEE Transactions on Information Theory, vol. 58, no. 4, 2280– 2302, 2012. [60] S. Landner and O. Milenkovic, “Algorithmic and Combinatorial Analysis of Trapping Sets in Structured LDPC Codes,” in International Conference on Wireless Networks, Communications and Mobile Computing, vol. 1, 630–635, IEEE, 2005. [61] X. Wang, Y. Zhang, S. Yu, and H. Guo, “High Speed Error Correction for Continuous-Variable Quantum Key Distribution with Multi-Edge Type LDPC Code,” Scientific Reports, vol. 8, no. 1, 1–7, 2018. [62] Y. Li, X. Zhang, Y. Li, B. Xu, L. Ma, J. Yang, and W. Huang, “HighThroughput GPU Layered Decoder of Quasi-Cyclic Multi-Edge Type Low Density Parity Check Codes in Continuous-Variable Quantum Key Distribution Systems,” Scientific Reports, vol. 10, no. 1, 1–11, 2020. [63] F. Grosshans, “Collective Attacks and Unconditional Security in Continuous Variable Quantum Key Distribution,” Physical Review Letters, vol. 94, 020504, 2005. [64] M. Navascu´es, F. Grosshans, and A. Acin, “Optimality of Gaussian Attacks in Continuous-Variable Quantum Cryptography,” Physical Review Letters, vol. 97, no. 19, 190502, 2006. [65] S. Fossier, E. Diamanti, T. Debuisschert, R. Tualle-Brouri, and P. Grangier, “Improvement of Continuous-Variable Quantum Key Distribution Systems by Using Optical Preamplifiers,” Journal of Physics B: Atomic, Molecular and Optical Physics, vol. 42, no. 11, 114014, 2009. [66] P. Jouguet, S. Kunz-Jacques, E. Diamanti, and A. Leverrier, “Analysis of Imperfections in Practical Continuous-Variable Quantum Key Distribution,” Physical Review A, vol. 86, no. 3, 032309, 2012.

Optimised Multithreaded CV-QKD Reconciliation for Global Quantum Networks: Conclusion & References

This paper is available on arxiv under CC 4.0 license. Authors: (1) Xiaoyu Ai, School of Electrical Engineering & Telecommunications, University of New South Wales, Sydney, NSW 2052, Australia; (2) Robert Malaney, School of Electrical Engineering & Telecommunications, University of New South Wales, Sydney, NSW 2052, Australia. Table of Links Abstract & Introduction System Overview Overcoming the limitation of key reconciliation Final Key Rate Experimental Results Discussion Conclusion & References Appendix Abstract Designing a practical Continuous Variable (CV) Quantum Key Distribution (QKD) system requires an estimation of the quantum channel characteristics and the extraction of secure key bits based on a large number of distributed quantum signals. Meeting this requirement in short timescales is difficult. On standard processors, it can take several hours to reconcile the required number of quantum signals. This problem is exacerbated in the context of Low Earth Orbit (LEO) satellite CV-QKD, in which the satellite flyover time is constrained to be less than a few minutes. A potential solution to this problem is massive parallelisation of the classical reconciliation process in which a large-code block is subdivided into many shorter blocks for individual decoding. However, the penalty of this procedure on the important final secured key rate is non-trivial to determine and hitherto has not been formally analysed. Ideally, a determination of the optimal reduced block size, maximising the final key rate, would be forthcoming in such an analysis. In this work, we fill this important knowledge gap via detailed analyses and experimental verification of a CV-QKD sliced reconciliation protocol that uses large block-length low-density parity-check decoders. Our new solution results in a significant increase in the final key rate relative to non-optimised reconciliation. In addition, it allows for the acquisition of quantum secured messages between terrestrial stations and LEO satellites within a flyover timescale even using off-the-shelf processors. Our work points the way to optimised global quantum networks secured via fundamental physics. I. INTRODUCTION Continuous Variable (CV) Quantum Key Distribution (QKD) has been intensively studied and significant breakthroughs have been achieved in both theory and experiment (see [1] for review). Compared to Discrete Variable (DV) QKD [2]–[5], CV-QKD can be implemented with well-developed technologies (e.g., homodyne detectors) in commercial fibreoptic networks [6], [7] and free-space optical communications [8], [9], providing it a potential advantage in practical deployments [10]–[14]. Considering the finite-key security of CV-QKD and DV-QKD, there are three critical parameters. These are, No, the number of original quantum signals sent by the transmitter (Alice) that are collected by the receiver (Bob); Ne, the number of quantum signals from which the protocol parameters are estimated;[1] and, ǫ, the probability that a QKD protocol fails to generate secret keys [15], [16]. To satisfy an upper limit on the failure probability of parameter estimation, Alice and Bob set Ne to a large value, which in turn implies a larger No. Despite the advantages in deployment, CV-QKD systems tend to demand a larger No to reach the same ǫ relative to DVQKD protocols. For example, to achieve a final key rate of 0.1 bits per pulse with ǫ = 10−9 , a CV-QKD protocol studied in [17] required No ≈ 109 signals. However, to achieve the same final key rate with ǫ = 10−14, the DV-QKD protocol in [18] required No ≈ 104 signals. This higher number of required signals in CV-QKD can render the classical post-processing (i.e. key reconciliation and privacy amplification[2] ) slow - possibly failing to meet target timescales for reconciliation. The end-users of a CV-QKD system expect the system to deliver two identical and secure keys under a limited time interval. For example, for satellite-based deployments, we would hope that the reconciliation is completed while maintaining a line-of-sight connection with the ground station. For a CV-QKD-enabled satellite with orbital parameters similar to Micius [20], this would mean the reconciliation should be completed in less than a few minutes. For the protocol we use in this work (see later), and for ǫ = 10−9 , this, in turn, would require the data rate of reconciliation to be at least 3.6 × 106 bits per second. For real-time reconciliation (say in subsecond timescales), two orders of magnitude increases in the reconciliation rates would be required. Demands for smaller ǫ will exacerbate the issue. Ideally, the rate of reconciliation should always be faster than the rate of quantum signalling. This all raises the question as to whether current CV-QKD reconciliation schemes are optimised for the highest possible key rates in bits per second. As we show here, this is not the case. Further optimisation is possible on all current schemes. To understand the issue better, we define reconciliation in the context of CV-QKD as a two-step scheme where the inputs to the reconciliation are non-identical N = 2No − 2Ne quadrature values[3] held by Alice and Bob (after parameter estimation), and the output is an identical bit string held by Alice and Bob [21]–[23]. Assuming a reverse reconciliation scheme, Bob first converts the quadrature values encoded by Alice in each signal to m bits. Alice, after converting each of her encoded real numbers also to m bits, then initiates some discrepancy-correction algorithms based on pre-defined errorcorrection codes to ensure her mN bits are identical to Bob’s. In this work, we will adopt Low-Density Parity-Check (LDPC) codes for the error correction. However, as alluded to above, reconciling mN bits within a limited time frame can be challenging. State-of-the-art LDPCbased reconciliation schemes for CV-QKD systems involve parallelised computation on a Graphics Processing Unit (GPU) [13], [24] or Field-Programmable Gate Arrays (FPGAs) [25], [26]. Reconciliation schemes implemented on FPGAs offer more programmable flexibility, but sometimes at the cost of reduced memory access relative to GPUs. For our purposes, both hardware architectures are useful - both offer massive parallelisation opportunities. These parallelisation solutions generally take the following two-step approach: 1) The mN bits are organised as m N-bit blocks to be reconciled. Each N-bit block is divided into multiple shorter blocks of size, say, NR. This is usually just set to a block size that can be processed within some timescale. 2) Then the m NR-bit blocks are reconciled in parallel (via independent processors) using optimally-designed LDPC decoders. However, what is missing in this approach is a proper optimisation analysis as to what the optimal value of NR is. As we show below, simply reducing NR at the cost of additional processing units is not an optimal solution. It transpires that in QKD the “penalty” cost of reducing the code rate (implicit in the use of small block lengths) significantly influences the bit per second final key rate. A more sophisticated analysis is required to determine the optimal reduced block length. Such an analysis is the key contribution of this work. Although we will adopt a specific CV-QKD protocol for our analysis, the key steps of our scheme will apply to any CV-QKD protocol. Our reconciliation scheme will deliver the highest reconciliation rate for a given processor speed - thus allowing for the optimal solution to CV-QKD reconciliation. [1] More precisely, in a CV-QKD protocol, Alice and Bob randomly select a Ne-signal subset from the No signals to estimate the parameters. [2] In this work, we focus on the key reconciliation step because it is the more time-consuming part in the post-processing steps while the privacy amplification involving only bit-wise operations can be easily implemented faster than the reconciliation [19]. [3] No and Ne are multiplied by 2 since Alice and Bob utilise both quadratures from heterodyne detection - the detection process we assume in this work.

Optimised Multithreaded CV-QKD Reconciliation for Global Quantum Networks: Abstract & Introduction

Life happily carries on and we don’t even have to look at our own darkness. Perhaps the real problem with modern life is the benign micro-systems that foster the covert conspiracy- we agree to blame the world for things that are, in fact, within us all. Implicitly we agree to live in a superficial dystopia of masks and make-up, where the wounds of psyche and identity stay hidden below the surface. The very comforts that our system affords us, also engage us in a conspiracy of denial, where we avoid the discomfiting process of taking a deep look within, and consequently, we allow the demons free space in our psyche. That’s why the unacknowledged, negative parts of us don’t simply lay dormant; they affect our thinking and our behavior, both conscious and unconscious, in ways that can be insidious and even destructive. Recognizing our shadows is a positive expression of our autonomy, an alternative to the easy way of least resistance, it is a refusal to ignore parts of ourselves, and a positive act of contact with that essential but compromised figure we all carry around inside our heads in the form of ‘I’. Recognition, then, is neither masochism nor resigned submission to this feature of our psychology. It is a positive move towards genuine self-awareness, which – by integrating in this way the previously separate shadow, by bringing that shadow into the full light of consciousness – is then powerless to continue the task it set itself, i.e, to deceive us into projecting onto other people the pain or injustice it embodies. No longer can we be unsuspecting hosts for the behaviors that are put to use by the shadow with such vicious selectivity. It takes courage and defiance to engage in this process of recognition – both externally and inwardly – to challenge how confinement insulates us from the truth of our ambivalence, how confinement requires people to live in a suspended way, aware of only part of themselves. If we allow ourselves to acknowledge the full range of our humanity, we open ourselves to a greater capacity for change. There is power in the acknowledgment of our demons, which changes their ability to function in the dark; it will not make us immune to them, but it should let us manage them better. Knowing that we have a dark side expands our compassion towards ourselves and others. We all are fighting a battle within, and this makes us vulnerable to each other. But awareness can transform vulnerability into a source of solidarity. In fact, this recognition opens the door for a more equitable and empathetic society. In owning our inner conflict, we reduce the risk of projecting it onto others. This creates space for a culture of accountability and a compassionate conversation with our shadow as an integral part of the work of being in this world. It transforms the myth of ‘what was done’ to ‘what I’ve done’, and perhaps becomes a lifelong conversation with our courageous and compassionate inner voice: ‘This is what was done to me. Now, what can be done with it?’ To opt out of a system that not only permits but welcomes this denial of our demons is an act of profound individual and societal autonomy. It serves as an affirmation that we’re no longer willing to be accomplices to lying to ourselves and others; and, perhaps more vitally, that we decline to be complicit in the injustice of the status quo. Then, armed with this existential awareness of who we really are, we can begin to chart a course for a different kind of beings. We can let our actions become less reactive and more reflective, our behaviors less governed by dark assumptions, and conscious biases – and more the results of a holistic sense of self. In this emerging reality, autonomy and identity are no longer ideals we strive to achieve but realities we actively embody. They are a commitment to being human with grace, intentionality, and an intense dedication to becoming more fully human as individuals and a larger society. If we and create systems that nurture our deepest values, we unshackle ourselves from the painful inescapability of separate selves, and liberate ourselves to co-create a world in which human beings live their differences together in safety and dignity. pay attention to our demons,

Can Humanity's Systems Serve as Material Enablers for Evasion?

This paper is available on arxiv under CC 4.0 license. Authors: (1) Terence Blésin, Institute of Physics, Swiss Federal Institute of Technology Lausanne (EPFL) & Center of Quantum Science and Engineering (EPFL); (2) Wil Kao, Institute of Physics, Swiss Federal Institute of Technology Lausanne (EPFL) & Center of Quantum Science and Engineering (EPFL); (3) Anat Siddharth, Institute of Physics, Swiss Federal Institute of Technology Lausanne (EPFL) & Center of Quantum Science and Engineering (EPFL); (4) Alaina Attanasio, OxideMEMS lab, Purdue University; (5) Hao Tian, OxideMEMS lab, Purdue University; (6) Sunil A. Bhave, OxideMEMS lab, Purdue University; (7) Tobias J. Kippenberg, Institute of Physics, Swiss Federal Institute of Technology Lausanne (EPFL) & Center of Quantum Science and Engineering (EPFL). Table of Links Introduction Results A. Physics and design B. Device characterization C. Bidirectional microwave-optical transduction D. Pulsed transduction Discussion Acknowledgments Appendix A: Theory Appendix B: Fabrication process flow Appendix C: Optical characterization Appendix D: Characterization of acoustic resonances Appendix E: Characterization of bidirectional transduction Appendix F: Optimization of optical extraction efficiency Coherent interconversion between microwave and optical frequencies can serve as both classical and quantum interfaces for computing, communication, and sensing. Here, we present a compact microwave-optical transducer based on monolithic integration of piezoelectric actuators atop silicon nitride photonic circuits. Such an actuator directly couples microwave signals to a high-overtone bulk acoustic resonator defined by the suspended silica cladding of the optical waveguide core, which leads to enhanced electromechanical and optomechanical couplings. At room temperature, this triply resonant piezo-optomechanical transducer achieves an off-chip photon number conversion efficiency of 1.6 × 10−5 over a bandwidth of 25 MHz at an input pump power of 21 dBm. The approach is scalable in manufacturing and, unlike existing electro-optic transducers, does not rely on superconducting resonators. As the transduction process is bidirectional, we further demonstrate synthesis of microwave pulses from a purely optical input. Combined with the capability of leveraging multiple acoustic modes for transduction, the present platform offers prospects for building frequency-multiplexed qubit interconnects and for microwave photonics at large. I. INTRODUCTION Modern data centers have seen rapidly increasing traffic that motivates an overhaul of existing network infrastructures. As optical fibers support nearly lossless transport and high bandwidths, twisted-pair copper deployment has shrunk in favor of optical interconnects. Energy-efficient optical transceivers [1] and optical network architectures [2] are being explored in parallel to accommodate emerging data- and resource-intensive applications. In an analogous fashion, processing quantum information in superconducting circuits and networking via photonic interconnects has been envisioned as an effective strategy to address the scalability challenges in advancing quantum technologies [3–5]. The scheme, featuring the full universal set of microwave quantum gates [6] with vanishing thermal occupancy and loss of the optical channels, calls for the development of microwaveoptical transducers to bridge the two energy scales that differ by more than four orders of magnitude. Aside from facilitating the networking of remote quantum processors, these transducers may enable fully optical control and readout of microwave qubits [7, 8]. The subsequent replacement of coaxial lines bridging room temperature and cryogenic environments by optical fibers is expected to significantly ease the space and heat load constraints in dilution refrigerators, opening up a path toward upscaling processor units housed in a single fridge. Efficient frequency conversion requires a nonlinear interaction stronger than the coupling to loss channels. The highest conversion efficiency to date has been achieved using an electro-optomechanical approach [9]. There, a silicon nitride membrane interacts with an optical mode of a free-space Fabry-Pérot cavity via radiation pressure and simultaneously serves as the top plate of a capacitor, parametrically coupling the mechanics to the microwave resonator. By virtue of the high resonator quality factors and pump power handling capability, 47% of input photons can be interconverted between optical and microwave domains [10], approaching the 50% efficiency required for attaining finite quantum capacity [11]. This highly efficient transducer has enabled optical dispersive qubit readout with negligible excess backaction [12]. The low-frequency (MHz) mechanical intermediary nevertheless limits the transduction bandwidth and leads to appreciable added noise even at dilution refrigerator temperature. To address these drawbacks, piezo-optomechanical transducers based on optomechanical crystals (OMC) have been developed [13–22], where a tightly confined high-frequency (GHz) mechanical mode and a co-localized optical mode can interact at a vacuum optomechanical coupling rate g0 ∼ 2π × 500 kHz. The trade-off lies instead in the sophistication required for microwave-phonon wave matching that significantly increases design complexity, as well as thermo-optic instability that constrains the intra-cavity photon number. An on-chip efficiency of 5% has recently been reported on such a platform [20]. However, low thermal conductance of these suspended quasi-one-dimensional structures hinders correlated microwave-optical photon pair generation at a practical rate. Cavity electro-optic modulators constitute a conceptually simpler approach where Pockels effect directly mediates microwave-optical interaction [23–25]. On-chip realizations employing planar superconducting microwave resonators, benefited from the deep sub-wavelength mode volume of the vacuum electric field, to reach single photon electro-optic coupling rates g0 ∼ 2π × 1 kHz [26– 29]. However, the material science of χ (2) crystals poses additional challenges. Photorefractive effects—observed, for instance, in LiNbO3 thin films [30]—hamper optical power handling, while piezoelectric loss and scattered optical photons degrade the quality factor of superconducting resonators [31, 32]. The difficulty of producing smooth sidewall surfaces in the workhorse Pockels material, LiNbO3 , through dry etching results in propagation losses an order of magnitude above the absorption limit [33]. As a result, the maximum on-chip efficiency achieved with integrated electro-optic transducers exceeds just 2% [26]. In the spirit of Ref. [9], a bulk transducer comprising a mm-size mechanically polished LiNbO3 whispering gallery mode (WGM) resonator coupled to a 3D superconducting microwave cavity has proved competitive, trading g0 for improved power handling and quality factors [34, 35]. Taking one step further with pulsed optical pumping, the system has demonstrated not only a hallmark 14.4% total efficiency but also electro-optic dynamical backaction [36, 37] and microwave-optical quadrature entanglement [38]. With the aforementioned design trade-offs in mind, we present a new piezo-optomechanical transducer based solely on wafer-scale, CMOS-compatible fabrication processes. We utilize a low-loss silicon nitride (Si3N4 ) photonic molecule, as well as multiple GHz high-overtone bulk acoustic resonances (HBAR) parametrically coupled to the optical modes. Endowed with power handling capabilities superior to existing integrated transducers, the device fits compactly within a 100 µm-by-50 µm footprint, in contrast to state-of-the-art bulk designs [10, 36]. In addition, the transduction HBAR modes are more readily coupled to the microwave signal, unlike OMC transducers. We demonstrate bidirectional microwave-optical transduction with a bandwidth of 25 MHz and total efficiency up to 1.6 × 10−5 by pumping with 21 dBm of off-chip optical power in continuous-wave (CW) operation. The device is also characterized with a pulsed optical pump, which constitutes the first step toward optical control and readout of qubits, as well as heralded microwave-optical photon pair generation. The simple design, ease of fabrication, robust operation, and compact form factor anticipate wide applicability in quantum technologies and microwave photonics at large.

Bidirectional Microwave-optical Transduction Based on Integration: Introduction

Discussing the Impact of Our Exoplanet Study: How This Impacts Everything

This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Flor Ortiz, University of Luxembour; (2) Juan A. Vasquez-Peralvo, University of Luxembour; (3) Jorge Querol, University of Luxembour; (4) Eva Lagunas, University of Luxembour; (5) Jorge L. Gonzalez Rios, University of Luxembour; (6) Marcele O. K. Mendonc¸a, University of Luxembour; (7) Luis Garces, University of Luxembour; (8) Victor Monzon Baeza, University of Luxembour; (9) Symeon Chatzinotas, University of Luxembou. Table of Links Abstract & Introduction System Model and Problem Statement Antenna Design and Training Data Generation Supervised Learning for Adaptive Beamforming Numerical Results Conclusions, Acknowledgment, and References III. ANTENNA DESIGN AND TRAINING DATA GENERATION A. Antenna Design For GEO missions, efficient antenna design is crucial due to substantial free space losses and stringent constraints on permissible losses. To address these challenges, we have developed an open-ended waveguide antenna as the unit cell antenna for this scenario, as explained in [8]. This antenna design offers significantly lower losses than alternative solutions like patch antennas or dielectric-based antennas. The antenna consists of three main components: the open-ended waveguide itself, the groove polarizer responsible for circular polarization, and the rectangular-to-circular transition for connecting to the distribution network. Simulation results, confirm that the antenna meets the requirements. It radiates within the required frequency band, emitting Left-Hand Circular Polarized (LHCP) waves with minimal cross-polarization. This is evident from the S11 parameter, below -10 dB, and the axial ratio measuring less than 3 dB within the intended frequency range [8] We consider the coverage area over the Earth’s surface to determine the number of elements required for the array antenna. In the context of Very High Throughput satellite missions, where small areas need to be covered to avoid channel link saturation, we set a minimum beam diameter of 260 km or a coverage area Ac = 53093 km2 when the satellite is at nadir. It’s worth noting that the coverage area can be adjusted based on specific beam requirements We then calculate the 3 dB antenna beamwidth θ−3dB required to illuminate this area. Using geometric relations and the satellite’s location, we estimate θ−3dB, which in this case is found to be 0.41◦ . However, this number of elements is impractical regarding space, cost, and power requirements, as each element requires an RF chain. To address this, subarrays are introduced. The number of subarray elements is determined to avoid grating lobes intersecting the antenna’s Field of View (FoV) over the Earth’s surface resulting in an element spacing of d = 3.5λ0. With this spacing, subarrays of 4×4 elements are employed, resulting in 36×36 RF chains B. Training Data Generation The approach chosen in this study is array thinning, which consists of selectively activating and deactivating elements to form the desired antenna beam, as illustrated in Figure 1. Array thinning allows precise control of the beamwidth, uniform power distribution among the elements in multi-beam scenarios and orientation of the radiation pattern in the desired direction by progressive phase shifting. To create the training data for this antenna array, a Genetic Algorithm (GA) was employed. The goal of the GA was to find the optimal set of active antenna elements while respecting specific constraints, such as beamwidth, SLL and EIRP. The GA optimization process iteratively refines the antenna element configurations, starting from an initial configuration and continuing until convergence or until a predefined maximum iteration limit is reached. It considers several performance parameters simultaneously, such as beamwidth, SLL and EIRP, making it particularly suitable for optimizing complex antenna systems such as the distributed reflector antenna. The resulting training dataset consists of 174,203 samples, each representing different antenna element configurations that meet the defined constraints. This dataset serves as a reference for evaluating antenna performance and training machine learning models in the later sections of this paper. The main purpose of this database is to establish correlations between beamforming array weights and critical system parameters, including beamwidth, SLL, and EIRP. For more detailed information on the generated database, including access to the dataset itself, see [9], where it is openly available.

Supervised Learning Based Real-Time Adaptive Beamforming On-board: Antenna Design and Training Data

File with code #!/usr/bin/env python3

import shutil

def check_disk_usage(disk, min_absolute, min_percent):
    """Returns True if there is enough free disk space, false otherwise."""
    du = shutil.disk_usage(disk)
    # Calculate the percentage of free space
    percent_free = 100 * du.free / du.total
    # Calculate how many free gigabytes
    gigabytes_free = du.free / 2**30
    if percent_free < min_percent or gigabytes_free < min_absolute:
        return False
    return True

# Check for at least 2 GB and 10% free
if not check_disk_usage("/", 2*2**30, 10):
    print("ERROR: Not enough disk space")
    return 1

print("Everything ok")
return 0 Command: $ cp disk_usage.py disk_usage_original.py 
$ cp disk_usage.py disk_usage_fixed.py 
# this throws an error
$ ./disk_usage_fixed.py Code output: File "./disk_usage_fixed.py", line 19

    return 1

    ^

SyntaxError: 'return' outside function File with code Adds at the beginning, then change to and to . import sys return 1 sys.exit(1) return 0 sys.exit(0) #!/usr/bin/env python3

import shutil
import sys

def check_disk_usage(disk, min_absolute, min_percent):
    """Returns True if there is enough free disk space, false otherwise."""
    du = shutil.disk_usage(disk)
    # Calculate the percentage of free space
    percent_free = 100 * du.free / du.total
    # Calculate how many free gigabytes
    gigabytes_free = du.free / 2**30
    if percent_free < min_percent or gigabytes_free < min_absolute:
        return False
    return True

# Check for at least 2 GB and 10% free
if not check_disk_usage("/", 2*2**30, 10):
    print("ERROR: Not enough disk space")
    sys.exit(1)

print("Everything ok")
sys.exit(0) Command: $ ./disk_usage_fixed.py Code output: ERROR: Not enough disk space File with code Change in previous file to .Because the code changes the value to gigabytes, so don't need to multiply by . check_disk_usage("/", 2*2**30, 10) check_disk_usage("/", 2, 10) gigabytes_free = du.free / 2**30 2**30 #!/usr/bin/env python3

import shutil
import sys

def check_disk_usage(disk, min_absolute, min_percent):
    """Returns True if there is enough free disk space, false otherwise."""
    du = shutil.disk_usage(disk)
    # Calculate the percentage of free space
    percent_free = 100 * du.free / du.total
    # Calculate how many free gigabytes
    gigabytes_free = du.free / 2**30
    if percent_free < min_percent or gigabytes_free < min_absolute:
        return False
    return True

# Check for at least 2 GB and 10% free
if not check_disk_usage("/", 2, 10):
    print("ERROR: Not enough disk space")
    sys.exit(1)

print("Everything ok")
sys.exit(0) Command: $ ./disk_usage_fixed.py Code output: Everything ok Command: $ diff -u disk_usage_original.py disk_usage_fixed.py > disk_usage.diff
$ cat disk_usage.diff Code output: --- disk_usage_original.py	2023-06-22 15:13:38.591579963 -0700

+++ disk_usage_fixed.py	2023-06-22 15:41:35.013023839 -0700

@@ -1,6 +1,7 @@

 #!/usr/bin/env python3

 

 import shutil

+import sys

 

 def check_disk_usage(disk, min_absolute, min_percent):

     """Returns True if there is enough free disk space, false otherwise."""

@@ -14,9 +15,9 @@

     return True

 

 # Check for at least 2 GB and 10% free

-if not check_disk_usage("/", 2*2**30, 10):

+if not check_disk_usage("/", 2, 10):

     print("ERROR: Not enough disk space")

-    return 1

+    sys.exit(1)

 

 print("Everything ok")

-return 0

+sys.exit(0) Command: $ patch disk_usage.py < disk_usage.diff Code output: patching file disk_usage.py Command: $ ./disk_usage.py Code output: Everything ok

Git Guide 2: A Practical Application of Diff and Patch

If you’re working with large databases in , this story will sound familiar. As your Postgres database keeps growing, your performance starts to decline, and you begin to worry about storage space—or, to be precise, how much you’ll pay for it. You love PostgreSQL, but there’s something you wish you had: a highly effective data compression mechanism. Postgres does have somewhat of a compression mechanism: 🍞. In this post, we’ll walk you through how Postgres TOAST works and the different TOASTing strategies. As much as we enjoy a good TOAST, we’ll discuss why this is not the kind of compression feature you need for reducing the storage footprint of modern large databases—and how, as the PostgreSQL enthusiasts that we are here at Timescale, we decided to build a more suitable compression mechanism for PostgreSQL, inspired by the columnar design of NoSQL databases. PostgreSQL TOAST What Is Postgres TOAST? Even if it might reduce the size of datasets, TOAST (The Oversized Attribute Storage Technique) is not your traditional data compression mechanism. To understand what TOAST is, we have to start by talking about . how data is stored in PostgreSQL Postgres’ units of storage are called pages, and these have a fixed size (8 kB by default). Having a fixed page size gives Postgres many advantages, namely its data management simplicity, efficiency, and consistency, but it comes with a downside: some data values might not fit within that page. This is where TOAST comes in. TOAST refers to the automatic mechanism that PostgreSQL uses to efficiently store and manage values in Postgres that do not fit within a page. To handle such values, Postgres TOAST will, by default, compress them using an internal algorithm. If, after compression, the values are still too large, Postgres will move them to a separate table (called the TOAST table), leaving pointers in the original table. As we’ll see later in this article, you can actually modify this strategy as a user, for example, by telling Postgres to avoid compressing data in a particular column. TOAST-able Data Types The data types that might be subjected to TOAST are primarily variable-length ones that have the potential to exceed the size limits of a standard PostgreSQL page. On the other hand, fixed-length data types, like , , or , are not subjected to TOAST since they fit comfortably within a page. integer float timestamp Some examples of data types that might be subjected to TOAST are: and json jsonb Large strings text and (If the length specified in is small enough, then values of that column might always stay below the TOAST threshold.) varchar varchar(n) varchar(n) storing binary data bytea Geometric data like and and PostGIS types like or path polygon geometry geography How Does Postgres TOAST Work? Understanding TOAST not only relates to the concept of page size but also to another Postgres storage concept: tuples. Tuples are rows in a PostgreSQL table. Typically, the TOAST mechanism kicks in if all fields within a tuple have a total size of over 2 kB approx. If you’ve been paying attention, you might be wondering, “Wait, but the page size is around 8 kB—why this overhead?” That’s because PostgreSQL likes to ensure it can store multiple tuples on a single page: if tuples are too large, fewer tuples fit on each page, leading to increased I/O operations and reduced performance. Postgres also needs to keep free space to fit additional operational data: each page stores not just the tuple data but also additional information for managing the data, such as item identifiers, headers, and transaction information. So, when the combined size of all fields in a tuple exceeds approximately 2 kB (or the TOAST threshold parameter, as we’ll see later), PostgreSQL takes action to ensure that the data is stored efficiently. TOAST handles this in two primary ways: PostgreSQL can compress the large field values within the tuple to reduce their size using a compression algorithm that we’ll cover later in this article. By default, if compression is sufficient to bring the tuple's total size below the threshold, the data will remain in the main table, albeit in a compressed format. Compression. If compression alone isn't effective enough to reduce the size of the large field values, Postgres moves them to a separate TOAST table. This process is known as "out-of-line" storage because the original tuple in the main table doesn’t hold the large field values anymore. Instead, it contains a "pointer" or reference to the location of the large data in the TOAST table. Out-of-line storage. We’re simplifying things slightly for this article— read the PostgreSQL documentation for a full detailed view. The Postgres Compression Algorithm: pglz We’ve mentioned that TOAST can compress large values in PostgreSQL. But which compression algorithm is PostgreSQL using, and how effective is it? The (PostgreSQL Lempel-Ziv) is the default internal compression algorithm used by PostgreSQL specifically tailored for TOAST. pglz Here’s how it works in very simple terms: tries to avoid repeated data. When it sees repeated data, instead of writing the same thing again, it just points back to where it wrote it before. This "avoiding repetition" helps in saving space. pglz As reads through data, it remembers a bit of the recent data it has seen. This recent memory is referred to as the "sliding window." pglz As new data comes in, checks if it has seen this data recently (within its sliding window). If yes, it writes a short reference instead of repeating the data. pglz If the data is new or not repeated enough times to make a reference shorter than the actual data, just writes it down as it is. pglz When it's time to read the compressed data, uses its references to fetch the original data. This process is quite direct, as it looks up the referred data and places it where it belongs. pglz doesn't need separate storage for its memory (the sliding window); it builds it on the go while compressing and does the same when decompressing. pglz This implementation is designed to offer a balance between compression efficiency and speed within the TOAST mechanism. In terms of compression rate, the effectiveness of will largely depend on the nature of the data. pglz For example, highly repetitive data will compress much better than high entropy data (like random data). You might see compression ratios in the range of 25 to 50 percent, but this is a very general estimate—results will vary widely based on the exact nature of the data. Configuring TOAST TOAST strategies By default, PostgreSQL will go through the TOAST mechanism according to the procedure explained earlier (compression first and out-of-line storage next, if compression is not enough). Still, there might be scenarios where you might want to fine-tune this behavior on a per-column basis. PostgreSQL allows you to do this by using the TOAST strategies , , , and . PLAIN EXTERNAL EXTENDED MAIN : This is the default strategy. It implies the data will be stored out of line in a separate TOAST table if it’s too large for a regular table page. Before moving the data to the TOAST table, it will be compressed to save space. EXTENDED : This strategy tells PostgreSQL to store the data for this column out of line if the data is too large to fit in a regular table page, and we’re asking PostgreSQL not to compress the data—the value will just be moved to the TOAST table as-is. EXTERNAL : This strategy is a middle ground. It tries to keep data in line in the main table through compression; if the data is definitely too large, it'll move the data to the TOAST table to avoid an error, but PostgreSQL won't move the compressed data. Instead, it’ll store the value in the TOAST table in its original form. MAIN : Using in a column tells PostgreSQL to always store the column's data in line in the main table, ensuring it isn't moved to an out-of-line TOAST table. Take into account that if the data grows beyond the page size, the will fail because the data won’t fit. PLAIN PLAIN INSERT If you want to inspect the current strategies of a particular table, you can run the following: \d+ your_table_name You'll get an output like this: => \d+ example_table
                     Table "public.example_table"
 Column  |       Data Type   | Modifiers | Storage  | Stats target | Description 
---------+------------------+-----------+----------+--------------+-------------
  bar    | varchar(100000)  |           | extended |              | If you wish to modify the storage setting, you can do so using the following command: -- Sets EXTENDED as the TOAST strategy for bar_column 
ALTER TABLE example_blob ALTER COLUMN bar_column SET STORAGE EXTENDED; Key parameters Apart from the strategies above, these two parameters are also important to control TOAST behavior: TOAST_TUPLE_THRESHOLD This is the parameter that sets the size threshold for when TOASTing operations (compression and out-of-line storage) are considered for oversized tuples. As we’ve mentioned previously, by default, is set to approximately 2 kB. TOAST_TUPLE_THRESHOLD TOAST_COMPRESSION_THRESHOLD This is the parameter that specifies the minimum size of a value before Postgres considers compressing it during the TOASTing process. If a value surpasses this threshold, PostgreSQL will attempt to compress it. However, just because a value is above the compression threshold, it doesn't automatically mean it will be compressed: the TOAST strategies will guide PostgreSQL on how to handle the data based on whether it was compressed and its resultant size relative to the tuple and page limits, as we’ll see in the next section. Bringing it all together is the trigger point. When the size of a tuple's data fields combined exceeds this threshold, PostgreSQL will evaluate how to manage it based on the set TOAST strategy for its columns, considering compression and out-of-line storage. The exact actions taken will also depend on whether column data surpasses the : TOAST_TUPLE_THRESHOLD TOAST_COMPRESSION_THRESHOLD (default strategy): If a tuple's size exceeds , PostgreSQL will first attempt to compress the oversized columns if they also exceed . If compression brings the tuple size under the threshold, it will remain in the main table. If it doesn't, the data will be moved to an out-of-line TOAST table, and the main table will contain pointers to this external data. EXTENDED TOAST_TUPLE_THRESHOLD TOAST_COMPRESSION_THRESHOLD : If the tuple size surpasses the , PostgreSQL will try to compress the oversized columns (provided they're over the ). If compression allows the tuple to fit within the main table's tuple, it remains there. If not, the data is moved to the TOAST table in its uncompressed form. MAIN TOAST_TUPLE_THRESHOLD TOAST_COMPRESSION_THRESHOLD : PostgreSQL skips compression, regardless of the . If the tuple's size is beyond the , the oversized columns will be stored out-of-line in the TOAST table. EXTERNAL TOAST_COMPRESSION_THRESHOLD TOAST_TUPLE_THRESHOLD : Data is always stored in the main table. If a tuple's size exceeds the page size (due to having very large columns), an error is raised. PLAIN Strategy Compress if tuple > TOAST_COMPRESSION_THRESHOLD Store out-of-line if tuple > TOAST_TUPLE_THRESHOLD Description EXTENDED Yes Yes Default strategy. Compresses first, then checks if out-of-line storage is needed. MAIN Yes Only in uncompressed form Compresses first, and if still oversized, moves to TOAST table without compression. EXTERNAL No Yes Always moves to TOAST if oversized, without compression. PLAIN No No Data always stays in the main table. If a tuple exceeds the page size, an error occurs. Why TOAST Isn't Enough as a Data Compression Mechanism in PostgreSQL By now, you’ll probably understand why TOAST is not the data compression mechanism you wish you had in PostgreSQL. Modern applications imply large volumes of data ingested daily, meaning databases (over)grow quickly. Such a problem was not as prominent when our beloved Postgres was built decades ago, but today’s developers need compression solutions for reducing the storage footprint of their datasets. While TOAST incorporates compression as one of its techniques, it's crucial to understand that its primary role isn't to serve as a database compression mechanism in the traditional sense. TOAST is mainly a solution to one problem: managing large values within the structural confines of a Postgres page. While this approach can lead to some storage space savings due to the compression of specific large values, its primary purpose is not to optimize storage space across the board. For example, if you have a 5 TB database made up of small tuples, TOAST won’t help you turn those 5 TB into 1 TB. While there are parameters within TOAST that can be adjusted, this won't transform TOAST into a generalized storage-saving solution. And there are other inherent problems with using TOAST as a traditional compression mechanism in PostgreSQL, for example: Accessing TOASTed data can add overhead, especially when the data is stored out of line. This becomes more evident when many large text or other TOAST-able data types are frequently accessed. TOAST lacks a high-level, user-friendly mechanism for dictating compression policies. It’s not built to optimize storage costs or facilitate storage management. TOAST's compression is not designed to provide especially high compression ratios. It only uses one algorithm ( ) with compression rates varying typically from 25-50 percent. pglz Adding Columnar Compression to PostgreSQL With Timescale , PostgreSQL users have a better alternative. Inspired by the compression design of NoSQL databases, . This transformative approach transcends PostgreSQL’s conventional row-based storage paradigm, introducing the efficiency and performance of columnar storage. Via the TimescaleDB extension we added columnar compression functionality to PostgreSQL By adding a compression policy to your large tables, . you can reduce your PostgreSQL database size by up to 10x (achieving +90 percent compression rates) By defining a time-based compression policy, you indicate when data should be compressed. For instance, you might choose to compress data older than seven (7) days automatically: -- Compress data older than 7 days
SELECT add_compression_policy('my_hypertable', INTERVAL '7 days'); Via this compression policy, Timescale will transform the table ( ) into a columnar format behind the scenes, combining many rows (1,000) into an array. To boost compressibility, Timescale will apply different compression algorithms depending on the data type: partitions which in Timescale are also created automatically Gorilla compression for floats Delta-of-delta + with compression for timestamps and other integer-like types Simple-8b run-length encoding Whole-row dictionary compression for columns with a few repeating values (+ LZ compression on top) LZ-based array compression for all other types This columnar compression design offers an efficient and scalable solution to the problem of large datasets in PostgreSQL. It allows you to use less storage to store more data without hurting your query performance (it improves it). And in the latest versions of TimescaleDB, you can also , , and directly over compressed data. INSERT DELETE UPDATE Wrap-Up We hope this article helped you understand that while TOAST is a well-thought-out mechanism to manage large values within a PostgreSQL page, it’s not effective for optimizing database storage use within the realm of modern applications. If you’re looking for effective data compression that can move the needle on your storage savings, give Timescale a go. You can try our cloud platform that propels PostgreSQL to new performance heights, making it faster and fiercer— —or you can add to your self-hosted PostgreSQL database. it’s free, and no credit card is required the TimescaleDB extension Written by . Carlota Soto

Postgres TOAST: Understanding the Data Compression Mechanism and Its Limitations 

পোস্টগ্রেস টোস্ট: ডেটা কম্প্রেশন মেকানিজম এবং এর সীমাবদ্ধতা বোঝা

Postgres TOAST: Den Datenkomprimierungsmechanismus und seine Einschränkungen verstehen

Postgres TOAST: comprensión del mecanismo de compresión de datos y sus limitaciones

Postgres TOAST : Comprendre le mécanisme de compression des données et ses limites

पोस्टग्रेस टोस्ट: डेटा संपीड़न तंत्र और इसकी सीमाओं को समझना

Postgres TOAST: データ圧縮メカニズムとその制限について理解する

Postgres TOAST: 데이터 압축 메커니즘 및 한계 이해

Postgres TOAST: Compreendendo o mecanismo de compactação de dados e suas limitações

Postgres TOAST: понимание механизма сжатия данных и его ограничений

Postgres TOAST: Veri Sıkıştırma Mekanizmasını ve Sınırlamalarını Anlamak

Postgres TOAST: Tìm hiểu cơ chế nén dữ liệu và những hạn chế của nó

Postgres TOAST：了解数据压缩机制及其局限性

Step-by-Step Installation

In my , I have shown a way to unify the implementation of the most well-known graph-traversal algorithms. Now, let's make it more visually appealing and look into the performance differences. last post The story behind A few years ago, Yandex organized a contest called with an enticing prize: a ticket to a . The contest resembled a game, with participants tasked with finding optimal routes on a map and optimizing delivery using robotic couriers. Robots Couriers closed self-driving conference for professionals As I delved into the topic, I discovered that despite route finding being a solved problem, it continued to interest the professional game development community. Between 2010 and 2020, engineers , particularly beneficial for AAA games with massive maps. Reading on these optimizations was an exciting experience. made significant optimizations to the A* algorithm articles and research papers Furthermore, the contest requirements were designed to enable easy assessment of program outputs by the contest’s testing system. As a result, there was little emphasis on visualization. I found exploring this field intriguing and developing a small application that uses well-known graph algorithms to find routes on a grid map. To visualize my findings, I employed the SFML graphics library. Goal This project builds upon one of , where I demonstrated that four well-known path-finding algorithms (BFS, DFS, Dijkstra's, and A*) are not fundamentally different and can be implemented in a universal way. However, it was challenging to showcase significant performance differences among these algorithms in that project. my previous endeavors In this article, I aim to use improved test data and design something visually exciting. While the Yandex Contest task mentioned earlier aligns well with my goals, I will not solve their specific problem here since it heavily relies on their test system, which is currently unavailable. Instead, I will extract general ideas for input parameters from that contest and create my own implementation. Imaginary world Imagine a technically advanced and innovative city where the future has arrived long ago. In this city, the majority of orders are delivered by courier robots, and it has become a rarity for a person to deliver an order from a cafe. In this task, we invite you to participate in finding optimal routes to deliver orders efficiently. Let's envision the city as an N × N map. For simplicity, we assume that each robot occupies exactly one cell, and each cell can either be passable or not for the robots. In one step, a robot can move in any of the four cardinal directions (up, down, left, or right) if the target cell is free. And I'm of the original task: ignoring the rest At the beginning of the test, you need to output the number of robots you want to use to deliver orders and their initial coordinates. The construction of each robot will cost Costc rubles. Next, T iterations of the simulation will be performed. One iteration represents one virtual minute and consists of 60 seconds. At each iteration, your program will be given the number of new orders, and in response, the program should tell you what actions each robot performs (60 actions per robot). For each successfully delivered order, you will receive max(0, MaxTips - DeliveryTime) dollars in tips, where MaxTips is the maximum number of tips for one order, and DeliveryTime is the time from the moment the order appeared to its delivery in seconds. The total number of points that you earn in one test is calculated by the formula TotalTips - R × Costc, where TotalTips is the total number of tips earned, R is the number of robots used, Costc is the cost of building one robot. The Costc and MaxTips values are set in each test. If you earned less tips than you spent on making robots, your total points will be 0. You will also receive 0 points for the test if you perform any incorrect action. Input The program uses standard input to read the parameters. This approach allows us to specify test data of various complexities using input files. The first line of input contains three natural numbers: N (the size of the city map), MaxTips (the maximum number of tips per order), and Costc (the cost of building one robot). I ignored both MaxTips and Costc parameters for my first implementation and maybe will consider that in the future. Following that, each of the next N lines contains N characters representing the city map. Each string can consist of two types of characters: '#' - indicates a cell occupied by an obstacle. '.' - indicates a free space. Next, you will be provided with two natural numbers: T and D (T ≤ 100,000, D ≤ 10,000,000). T represents the number of interaction iterations, and D represents the total number of orders. Output Your task is to visualize the map and the optimal routes using the SFML graphics library. Modelling the maps I'm fond of maps represented as a grid of cells. Thus, I prefer to render all the results and map them as a grid on cell by cell basis. There is also the option to execute path search right on the grid without using any additional data structure (and I have implemented this as well for learning purposes: see in code). But because of a grid, it is easy to represent a map as a graph in one way or another. I prefer to use an adjacency list of cells for most of the algorithms like BFS, Dijkstra's, and A-Star. For algorithms like Bellman-Ford, it may make sense to use an Edges List instead of an Adjacency List. That's why if you explore the codebase, then you will find all of it, and they all are working examples. To split the logic and responsibility, I have a entity that is responsible for executing path finding according to the orders and tasks configuration that is specified via the file and related map files. Navigator App Config looks like this: App Config {
    "font": "../../data/arial.ttf",
    "map": "../../data/maps/test_29_yandex_weighten_real_map",
    "shadow": false,
    "map_": "../../data/maps/test_08_low_res_simple_map",
    "map__": "../../data/maps/test_10",
    "map___": "../../data/maps/test_07_partially_blocked_map",
    ... Note that "map_” "map__" etc. are not really configuration properties. They are ignored during the application run. Since there is no way to comment on the part of the JSON file, I use underlining in the property name so it can stay in the config file but not be used. looks like this: The map file 25 50 150
#########################
#########################
#########################
###.......#####.......###
###.......#####.......###
###.......#####.......###
###...................###
###.......#####.......###
###.......#####.......###
###...................###
######.###########.######
######.###########.######
######.###########.######
###.......#####.......###
###.......#####.......###
###.......#####.......###
###.......#####.......###
###.......#####.......###
###.......#####.......###
###.......#####.......###
######.###########.######
#########################
#########################
#########################
#########################
2 4
2
6 6 4 20 This is one of the simplest examples that contain either blocked or not blocked cells. I have prepared a lot of various examples of input parameters and test data. Starting from very small parts that let you debug and learn the code, finishing with a huge piece of map (from the real existing city) that allows us to measure the performance of a Graph algorithm. How do we draw such maps? When a map contains only cells with binary state (either blocked or non-blocked), this basically means that any edge of a graph either exists or not. To find a path in the graph, we have to represent it efficiently. Like in my previous post, I have used an with the relationship as Vector[NodeId]->points to->Vector[Neighbour Nodes]: adjacency list typedef std::vector<std::vector<std::shared_ptr<Cell>>> Graph; When we exploring grids it's not really necessary to use graphs at all. We are capable to traverse grid using BFS/DFS algorithms cell by cell without thinking about edges. See method for example. _GetPathByBFSOnGrid First, the initialization code reads the file and converts it into row by row and column by column: the grid bool RectangularMap::LoadMap(const std::string& filepath, bool shadow)
{
...
  // Fill the grid.
  _verticesNumber = 0;
  for (int row = 0; row < _height; row++)
  {
    ...
    for (int col = 0; col < _width; col++)
    {
      int x = col;
      int y = row;
      if (line[col] == BLOCK_CELL)
      {
        // Create a shared pointer here to safely pass pointers between the classes.
        _grid[row][col] = std::make_shared<Cell>(x, y, line[col], blockColor, shadow, _scaleFactor);
      }
      else
      {
        ...
      }
    }
  }

  // Make a graph
  InitialiseGraph();
...
} Then, it creates an actual graph as : an adjacency list void RectangularMap::InitialiseGraph()
{
  MapBase::InitialiseGraph();
  ...
  unordered_set<int> visited;

  for (int rr = 0; rr < _grid.size(); rr++)
  {
    for (int cc = 0; cc < _grid[rr].size(); cc++)
    {
      if (_grid[rr][cc]->GetId() > -1)
      {
        for (int i = 0; i < 4; i++)
        {
          int r = rr + dr[i];
          int c = cc + dc[i];

          if (r >= 0 && c >= 0 && r < _width && c < _height &&
              _grid[r][c]->GetId() > -1)
          {
            if (_isNegativeWeighten)
            {
              ...
            }
            else
            {
              _adjacencyList[_grid[rr][cc]->GetId()].push_back(_grid[r][c]);
            }
          }
        }
      }
    }
  }
} Grid representation is useful to draw on the screen using the SFML library. We can draw (this is exactly what I do for small maps): by creating a geometric object ...
for (int j = _visibleTopLeftY; j < _visibleBottomRightY; j++)
{
  for (int i = _visibleTopLeftX; i < _visibleBottomRightX; i++)
  {
    _grid[j][i]->Draw(_window, _scaleFactor);
  }
}
...
sf::RectangleShape tile;
tile.setSize(sf::Vector2f(_cellSize - 5, _cellSize - 5));
tile.setPosition(sf::Vector2f(_x * _cellSize, _y * _cellSize));
tile.setFillColor(_color);
window.draw(tile); Or we can do it for larger maps: efficiently pixel by pixel sf::Uint8* pixels = new sf::Uint8[_width * _height * 4];

for (int j = _visibleTopLeftY; j < _visibleBottomRightY; j++)
{
  for (int i = _visibleTopLeftX; i < _visibleBottomRightX; i++)
  {
    int index = (_grid[j][i]->GetY() * _width + _grid[j][i]->GetX());

    sf::Color color = _grid[j][i]->GetColor();
    pixels[index * 4] = color.r;
    pixels[index * 4 + 1] = color.g;
    pixels[index * 4 + 2] = color.b;
    pixels[index * 4 + 3] = color.a;
  }
}

sf::Texture texture;
texture.create(_width, _height);
texture.update(pixels);
sf::Sprite sprite;
sprite.setTexture(texture);
sprite.setScale(cellSize, cellSize);
_window.draw(sprite); Finally, let's see what a map defined by file would look like. Originally, the file defined this: test_25_xmax ..............C.................
..............#.................
.............###................
............#####...............
...........#######..............
..........##1###2##.............
.........###########............
........##3######4###...........
.......###############..........
......#################.........
.....###################........
....#####################.......
.............###................
.............###................
.............###................ And a map renderred with SFML looks like this: Because I wanted all of that to be controlled by the user with the keyboard, I left all the user-behavior logic in the . I like to call it “Controller” logic. main.cpp SFML library makes it very easy to handle keyboard events: while (window.isOpen())
{
  Event event;
  while (window.pollEvent(event))
  {
    if (event.type == Event::Closed)
      window.close();

    if (event.type == Event::KeyPressed && event.key.code == Keyboard::Space)
    {
      ... Do what you need here
    }
  }
} The main idea is for user triggers (press of a SPACE button) to read the map file and rendering the map, and then by a second trigger (second press of a SPACE button) to load the routing task and calculate the shortest path between two points on the map: ...
if (navigator->IsReady())
{
  navigator->Navigate(); // Finding route between two points
}
else
{
  if (map->IsReady()) // Second SPACE press runs the routing
  {
    skipReRendering = true;
    if (navigator->LoadTasks(filepath))
    {
      navigator->SetMap(map);
    }
  }
  else // Load and draw map
  {
    drawLoading(window, font);
    if (!map->LoadMap(filepath, shadowed))
    {
      return 0;
    }
    drawProcessing(window, font);
  }
}
... We need to go deeper I wanted to play with more Graph algorithms, and they all have their limitations, so I decided to implement multi-color maps that can be represented by multi-weighted graphs. Every cell is colored, and the color means that the edge not only exists but also applies some weight (or fee, or fine, you name it). So, the edge might be blocked, half blocked, not blocked, ... you got the idea. Thus, I have implemented multi-color maps that look more joyful and look like a game-ready (example from file ): test_31_multi_weight_graph_map Some of the configuration files contain more complex maps from really existing cities, like : test_29_yandex_weighten_real_map As a challenge, now we should handle maps with very flexible configuration. essentially contains a lot of logic inside, including all the graph algorithms and even more than needed (because I like to play with things, even if it's not particularly useful for now). RectangularMap.cpp I have implemented , , , algorithms, and a number of additional "utility" algorithms like Topological Sort, Single Source Path, that are not useful for the current application state (because they work on Directly Acyclic Graphs, which are not type of Graphs I currently use), but I have some ideas to use it in future improvements. BFS#Line 598 Dijkstra#Line 299 A-Star#Line 356 Bellman-Ford#Line 428 I didn't polish all the code and didn't get it ideal enough, but it allows me (and, I hope, will allow you) to play with the code and compare performance metrics. Sorry about some commented lines here and there, maybe some dirty code...it's all way of learning :). To grasp an idea of what's inside, I recommend you review the . RectangularMap.h There are also some fun features, like a Focus feature that allows one to render only a particular part of a map. It changes focus by re-rendering the necessary part using the when the user presses the PgDown or PgUp buttons. It is pretty easy to improve this feature and implement "Zoom" functionality. Take it as a homework, if you like it. Observer pattern Focus feature with map file in work: test_29_yandex_weighten_real_map Classes diagram looks like this: Run and play I believe the most joyful part is just running this little application is playing with variations of its configuration and algorithms. You can do a lot of experiments by using various map files as input parameters with different test data as well as change the logic in the code. After starting, you need to press SPACE an application will render a map according to the configuration file, and it makes a lot of sense to start exploring from the simplest test cases and moving forward to the most complex ones. Pressing SPACE one more time executes the routing algorithms and finds the path between the start and the nearest order. BTW, It's not yet done, but it is easy to implement reading all the rest of the orders available in map configuration files and execute pathfinding to all of them. Here is the route found on the map defined by file : test_18_yandex_super_high_res It is also capable of finding routes in the maps that simulate existing cities, like : test_29_yandex_weighten_real_map Finding efficient paths between two coordinates becomes challenging for algorithms like BFS but can be easily done by A-star. Based on the cells found in the map configuration files, the application will treat the map as a weighted or non-weighted graph and will select the right algorithm for it (and you can easily change this as well). It's easy to see the difference between BFS and A-Star performance: Final words With this, I want to leave you alone and let you play with these code examples. I hope you will find it fascinating and will learn a lot from it. Stay tuned! Links JPS+: Over 100x Faster than A* by Steve Rabin A* Optimizations and Improvements by Lucho Suaya https://hackernoon.com/bfs-dfs-dijkstra-and-a-star-are-basically-the-same-algorithm-ill-show-you-why My blog https://antonyarkov.substack.com/ Also published . here

Graph Traversal Algorithms: Visualizing Performance Variations in Route Finding Algorithms

Virtual reality (VR) and augmented reality (AR) are among the most significant advancements in the sphere of technology. These tools have evolved beyond their original entertainment purposes, now playing pivotal roles across various sectors, notably in training scenarios. Compared to conventional methods of training, VR-based training tools have gained a lot of traction. AR enhances our interaction with the real world by adding digital elements to it, facilitating a more enriching interaction with our environment. , on the other hand, immerses users in a purely digital environment, providing a comprehensive sensory experience. A few examples of VR training include flight drills, medical techniques, and complicated machine operations. VR The adoption of AR/VR in training modules is significant, given their ability to provide engaging and enveloping learning avenues unmatched by classical or online training approaches. Research that 26% of enterprises have adopted VR/AR for staff training, underscoring its rising acceptance in the business sector. indicates While traditional lecture-based and reading approaches yield retention rates of 5% and 10%, respectively, VR is a game-changer, securing a remarkable 75% retention. This effectiveness of VR training is only by the act of instructing others, which achieves an unparalleled 90% retention rate. rivaled The commercial prospects for AR and VR technologies are also worth noting. Forecasts from Statista the AR & VR sector's revenue to reach approximately US$31.12 billion by 2023, marking considerable expansion and prospects, spanning VR training modules and various other uses. anticipate Transitioning from Conventional Learning Methods to Engaging in VR Training Throughout time, traditional instructional techniques like lectures, reading materials, and hands-on demonstrations have dominated educational landscapes. While they've shown some efficacy, these methodologies often fall short in personalization, dynamic interaction, and instant feedback. As a result, these strategies generally provide a somewhat linear and passive learning experience, leading to innovations such as hands-on labs, collaborative learning sessions, and interactive workshops. The onset of the digital era introduced us to online education platforms, offering unprecedented flexibility and reach. These platforms not only facilitated distance learning but also placed a plethora of resources within learners' grasp. Yet, the craving for a deeper, more engaging learning experience persisted. With the rise of in training, we're witnessing a transformative movement. Coupled with AR, immersive VR training introduces fresh horizons for incredibly engaging and interactive educational settings. These profound virtual experiences seamlessly merge theoretical insights with real-world applications. VR and AR Advantages of AR/VR Training In the ever-changing world of education and professional growth, both augmented reality (AR) and virtual reality (VR) have surfaced as groundbreaking tools that transform the way we learn. By intertwining digital elements into training sessions, these captivating technologies open doors to a realm of interactive and meaningful training experiences. Enhancing Realistic Learning The introduction of AR and VR is set to redefine educational paradigms by immersing learners in lifelike virtual settings. Such digital recreations allow students to immerse themselves in practical experiences reflecting real-life challenges, paving the way for improved understanding and memory retention. Consider a medical trainee executing complex surgical steps in a safe digital space or military personnel undergoing tactical training without actual danger. The spectrum of hands-on learning is broad, with a profound effect on refining learners' skills. Reducing Real-World Risks A standout benefit of AR and VR training is its capacity to recreate perilous real-world scenarios without exposing trainees to inherent risks. This innovation is transformational for sectors prone to high risks, encompassing medical, defense, and heavy industries. By mimicking high-stakes situations, learners can hone their skills and judgment in a monitored setting, thereby cutting down the risk of mishaps when faced with real challenges. An insightful observation of VR's role in workplace safety is demonstrated in a study by Chaos Theory Games. Their findings revealed that VR training led to a remarkable 43% decrease in workplace accidents. This impressive figure highlights the potential of AR and VR in setting elevated safety benchmarks across diverse sectors. Bridging the Distance Gap With Remote Learning The dawn of AR and VR has initiated a progressive phase in distant learning, obliterating the constraints of location and granting broader access to top-notch training. Now, participants can immerse in dynamic training modules from their personal spaces, eradicating the necessity of being physically present at a specific training venue. This feature is a boon for those in secluded locations or those restricted from traveling for educational reasons. Moreover, this form of training paves the way for international teamwork and the sharing of insights, amalgamating a mix of expertise and viewpoints in a shared digital arena. Consequently, entities can cultivate an ethos of ongoing learning and talent enhancement, irrespective of their workforce's geographical dispersion. Customizing Learning Paths Generic training models often miss the mark in addressing the varied learning preferences and necessities of diverse individuals. This is where AR and VR stand out. These platforms allow educators to craft lessons specifically attuned to each student's distinct needs. By offering instant feedback and flexible learning modules, can gauge a student's strengths and highlight areas needing attention. Such an analytical approach lets educators refine their teaching techniques, ensuring optimal growth for every learner. AR and VR platforms Financial and Time Prudence While AR and VR systems might seem initially pricey, they usher in lasting savings in training duration, assets, and recurring costs for organizations. Substituting age-old training mechanisms that demand tangible assets and continuous expenditure, AR and VR emerge as financially prudent alternatives. AR/VR Training for Different Industries AR and VR technologies, by replicating real-life situations, present immersive learning experiences that harmonize theoretical concepts with real-world practice across diverse sectors. From healthcare to aviation and customer service, virtual training bolsters existing training paradigms. Healthcare In the healthcare sector, AR/VR has ushered in innovative learning avenues. Medical practitioners can now refine their surgical and diagnostic skills in a virtual, risk-free space, promoting heightened proficiency and fewer mistakes. These technologies not only enhance the skill sets of professionals but also cultivate confidence, ultimately profiting both patients and healthcare facilities. Recent surveys suggest a growing inclination among healthcare professionals toward adopting AR/VR-based training modules in the coming years. Emergency AR/VR technologies are invaluable in fields such as safety and emergency response. These tools offer realistic simulations of emergencies, allowing trainees to refine decision-making and response capabilities without facing real-world dangers. Police departments are increasingly turning to VR to hone skills ranging from de-escalation techniques to addressing implicit biases. Likewise, emergency response initiatives, like those in Austin, are using VR to prepare responders with ambulance operations and protocols, demonstrating that VR training can lead to improved retention and reduced errors. Manufacturing In sectors where intricate machinery and potential hazards are daily challenges, traditional training might fall short. AR/VR introduces immersive scenarios for machine operations, mitigating on-site risks while boosting output and effectiveness. This hands-on virtual approach paves the way for a proficient and confident workforce, as they can engage with virtual machinery and processes before handling actual equipment. Customer Service Quality customer interactions are vital for many businesses, and AR/VR tools are elevating soft skills training in this realm. Virtual environments simulate diverse customer scenarios, allowing staff to refine their communication, problem-solving, and empathy skills, leading to enhanced customer satisfaction. Automotive Industry The automotive sector is witnessing a renaissance with the integration of AR/VR. From design to assembly and even customer experience, these tools are proving indispensable. Virtual environments streamline the design process, enhance assembly proficiency, and offer technicians interactive repair guides. Prominent automotive brands like BMW employ mixed reality for efficient design processes, while Audi's VR training focuses on error reduction across intricate systems. Volkswagen leverages VR for versatile training options and collaborations, signifying the transformative role of AR/VR in this industry. Defense AR/VR technologies have transformed training and operations in the defense sector, offering realistic combat scenarios, improved mission planning, and aiding in the design of military equipment. Partnerships, such as the one between SimX and the US Air Force, showcase the advancements in medical training for troops, emphasizing the growing reliance on these technologies for more effective preparation. Logistics In logistics, AR/VR tools elevate employee training and optimize operational performance. Companies like DB Schenker harness VR for intricate tasks, such as forklift operations, by simulating realistic environments. Similarly, NS uses VR to train staff in tackling unforeseen situations at train stations, highlighting the utility of immersive simulations in enhancing efficiency. Finance , are changing the training landscape in finance. Firms like Bank of America are leveraging VR to mimic customer interactions, thereby training staff in tasks ranging from account management to navigating emotional scenarios, leading to increased efficiency and empathy. AR/VR technologies, coupled with AI Aviation The aviation sector's embrace of AR/VR is evident in its widespread adoption across companies. Boeing's VR training for NASA astronauts and Avietra's safety simulations for cabin crew members epitomize the industry's shift toward virtual training. Additionally, collaborations like that of Airbus, Air France, and KLM spotlight the potential of VR in aircraft maintenance. Rolls Royce and KLM further underscore the importance of VR in training mechanics and pilots, respectively, signaling a trend toward more immersive and efficient training methods. HoReCa In the HoReCa sector, AR/VR is redefining customer engagement and staff training. From AR-enhanced interactive menus to VR-driven kitchen simulations, these technologies are enriching guest experiences and streamlining operations. Instances like Honeygrow's adoption of VR for employee onboarding underline the potential of these tools in enhancing staff competency and understanding of operational processes. Key Challenges of AR/VR Training While AR and VR promise transformative and engaging learning experiences, they are not without their set of hurdles and shortcomings. Addressing these is paramount to unlocking their full potential in specialized training. Tackling Technological Hurdles AR and VR training is largely dependent on sophisticated hardware and software, occasionally leading to technological impediments. Unexpected technical snags and hardware constraints can interrupt the flow of training. However, the consistent progress in both hardware and software domains is making these technologies increasingly user-centric. As technology evolves, we can anticipate fewer glitches and a more streamlined AR and VR training experience. Tracking and Evaluating Trainee Development In AR/VR settings, monitoring trainee growth can be daunting, given the virtual ambiance of the session. Traditional assessment tools might fall short in this context. Yet, the integration of advanced data analytics and specific performance indicators into AR/VR systems offers a solution. These analytics shed light on trainee actions, choices, and information assimilation, equipping trainers with tools to personalize training content. Acceptance and Hesitancy Toward Technological Shift Transitioning to AR/VR necessitates a departure from age-old training methods. Such changes can be met with skepticism from trainers and students unfamiliar with or uncertain about these new tools. But, as the merits of AR and VR become evident, such reluctance should wane. Highlighting the tangible benefits – like heightened engagement, faster learning curves, and better memory retention – will foster a more widespread acceptance. Health Implications Prolonged exposure to AR/VR gadgets can sometimes cause minor health challenges like dizziness or ocular fatigue. These discomforts can act as deterrents to AR and VR training. To counteract these issues, manufacturers are emphasizing user-friendly designs and better hardware and software for devices. Also, punctuating sessions with breaks and offering customizable settings can further elevate the AR and VR training experience. Financial Aspects Rolling out VR training can be financially daunting. The combined costs of advanced hardware and software and the creation of top-tier AR/VR content might strain organizational budgets. Yet, as these technologies become more mainstream and market competition intensifies, cost factors should become more manageable, making such training more economically viable. Challenges in Accessibility A primary shortcoming in VR and AR training is that not all trainees can access the required equipment or online connectivity. This disparity might exclude some from the training process. A potential solution could be blending traditional training techniques with AR and VR or setting up dedicated training hubs equipped with the latest tech for all to use. Navigating Through a Lack of Set Norms The void in universal standards for AR and VR training can result in hiccups in content design and rollout. Without a set benchmark, developers might grapple with compatibility challenges across platforms. A practical approach is to design software tailored for specific hardware models or employ adaptable platforms, such as Unity or Unreal Engine. The Prospective Landscape of AR and VR Training The horizons of training and education are expanding rapidly with the integration of AR and VR technologies. When intertwined with the advanced capabilities of AI and ML, they pave the way for dynamic learning, insightful analytics, and intelligent process optimization. These cutting-edge technologies facilitate personalized, immersive learning experiences that promise enriched understanding and readiness for our increasingly digitalized world. Although current challenges, such as technological constraints and initial high costs, may seem daunting, the transformative advantages of AR and VR far surpass these obstacles. The prospect of providing learners with vibrant, three-dimensional, and interactive training experiences compels businesses and educational institutions alike to delve deeper into these technologies. By adopting AR and VR, we stand at the cusp of a groundbreaking shift in learning methodologies, ushering in an era of boundless exploration and unprecedented innovation in education and training. Also published . here

AR and VR Training: Leveraging Capabilities

এআর এবং ভিআর প্রশিক্ষণ: ক্ষমতার ব্যবহার

AR- und VR-Training: Möglichkeiten nutzen

Capacitación en AR y VR: aprovechamiento de las capacidades

Formation AR et VR : tirer parti des capacités

एआर और वीआर प्रशिक्षण: क्षमताओं का लाभ उठाना

AR および VR トレーニング: 機能の活用

AR 및 VR 교육: 기능 활용

Treinamento em AR e VR: aproveitando recursos

Обучение AR и VR: использование возможностей

AR ve VR Eğitimi: Yeteneklerden Yararlanma

Đào tạo AR và VR: Tận dụng khả năng

AR 和 VR 培训：利用能力

Today’s business requires accurate and timely data in order to make decisions. Data-driven applications and dashboards built on time series data are found in every enterprise. A great deal of business data is time series - from system and application metrics to weather, stock prices, network telemetry, IoT sensors, all rely on complex time series analysis and visualization to provide business insight. There are more than a few databases that are built for this kind of analysis, such as Clickhouse (for more on Clickhouse and MinIO see ), Snowflake, Amazon’s TimeStream and RedShift, and Apache Druid. A typical workflow involves gathering data, likely being streamed in real time or pulled from a data lake, transforming and ingesting it into a database, and then building applications and dashboards using SQL queries running against the database. Integrating ClickHouse with MinIO This blog post focuses on , specifically how to run Druid with MinIO as deep storage. We’ll go into more detail later, but in short Druid saves data to S3 deep storage and swaps it in and out of local memory as needed when running queries. We’re going to install MinIO, install Druid, edit some configuration files to tell Druid to use MinIO, and then import demo data and visualize it using . Apache Druid Apache Superset Prepare to be amazed at how quickly you can build a high-performance analytics and visualization stack with open source products. Druid Overview Druid is one of the fastest data stores around - it was built to combine the advantages of data warehouses, time series databases and search systems. Druid achieves tremendously speedy query responses by combining the key characteristics of these three systems into its ingestion layer, storage format, query layer and core architecture. Druid is typically deployed with other open source technologies such as Apache Kafka, Apache Flink and sits between a storage or processing layer and the end user, serving as a query layer for analytic workloads. To learn more about how Druid works, please see . Apache Druid 101 Druid is cloud-native and relies on a microservice-based architecture. Each core service in Druid (ingestion, query, coordination) can be deployed and scaled independently in containers or on commodity hardware. In addition, Druid services can independently fail and restart or be updated without impacting other services. Druid has several process types: processes manage data availability on the cluster. Coordinator processes control the assignment of data ingestion workloads. Overlord processes handle queries from external clients. Broker processes are optional; they route requests to Brokers, Coordinators, and Overlords. Router processes store queryable data. Historical processes ingest data. MiddleManager These processes are typically deployed in groups onto three different server or node types: : Runs Coordinator and Overlord processes, manages data availability and ingestion. Master : Runs Broker and optional Router processes, handles queries from external clients. Query : Runs Historical and MiddleManager processes, executes ingestion workloads and stores all queryable data. Data Druid has three major external dependencies: deep storage, metadata storage and ZooKeeper. Briefly, metadata is used to track segment usage and task information, and is usually handled by PostgreSQL or MySQL. Zookeeper is used for internal service discovery and coordination. As this is the MinIO blog, we’re going to focus on deep storage. Deep storage is a shared file system accessible by every Druid server where data is stored after it is ingested. In a single-server deployment this is simply a local drive. In clustered deployments, deep storage would take the form of a distributed object store like S3 or MinIO, HDFS or a network mounted filesystem. is how Druid transfers data between processes and backs up data. Druid stores data in files called segments. There’s an internal mapping that pulls segments from deep storage to cache them on local drives and in memory. In order to minimize latency, queries run against segments locally, not against segments stored in deep storage. Deep storage enables Druid’s elastic, fault tolerant design. Druid provides durability by bootstrapping from deep storage in the event that a data node goes down and comes back up. One caveat is that you’ll need enough free space on data nodes and in deep storage for the data you want to work with in Druid. Deep storage Druid uses the S3 API for object storage for deep storage. Using S3 API compatible object storage such as MinIO gives you the freedom to run Druid anywhere, which is something you might want to do to improve performance and security. MinIO is a superb deep storage for Druid because it has excellent S3 compatibility, the performance to meet requirements for real time OLAP and, perhaps most of all, the ability to deliver performance at scale. MinIO also provides necessary and . data protection lifecycle management You can run Druid, MinIO and Superset directly on Linux or in containers on Docker or Kubernetes. There are already some great resources out there that show you how to get Druid up and running, so if for some reason this blog post doesn’t satisfy you, check out for Linux, for Docker and for Kubernetes. Apache Druid Quickstart Docker · Apache Druid Clustered Apache Druid® on your Laptop Installing MinIO If you’re not already running MinIO, here are . The process is described thoroughly in . installation instructions The MinIO Quickstart Guide On Linux, for example, download MinIO and run it (feel free to change the root password - and remember it): wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
MINIO_ROOT_USER=admin MINIO_ROOT_PASSWORD=password ./minio server /mnt/data --console-address ":9001" Launch a browser and connect to MinIO Server at using the root user and password you configured earlier. http://<your-IP-address>:9000 The first time you log in, you’ll be taken to the Buckets page. Create the bucket . druidbucket In the real world, you would probably want to secure access to your Druid data, but we don’t have to do that in our lab. You would create a service account (under the Identity menu), edit the user policy to only allow access to and in the Druid configuration below use the service account’s access key and secret key. druidbucket Installing Druid The easiest way to test drive Druid is via Docker. Follow the instructions  and use this . This creates a container for each Druid service, as well as Zookeeper and PostgresSQL. This method uses an to specify the Druid configuration. Download to where you’ve saved Druid’s containers. For the instructions below (Configure Druid for MinIO), you can edit the environment file to pass the variables to standard files or edit the configuration files directly within the container. Docker · Apache Druid docker-compose.yaml environment file this file Druid configuration Start Druid with . Once the cluster comes up, open a browser and navigate to to use the Druid console. docker-compose up http://localhost:8888 Configure Druid for MinIO Now it’s time to configure Druid to use MinIO for deep storage. In this step, you will need to edit Druid config files (or the Docker environment files as described above) to reference MinIO and then restart Druid. The process is fully described in and we’re including an abbreviated version here for quick reference. How to Configure Druid to Use Minio as Deep Storage - DZone Big Data Depending on how you’ve installed Druid, In the file, add to . Add extensions by name, in quotes and separated by a comma, for example: conf/druid/_common/common.runtime.properties "druid-s3-extensions" druid.extensions.loadList druid.extensions.loadList=["druid-parser-route", "druid-s3-extensions"] According to the above referenced article, the S3 extension for deep storage relies on jets3t under the hood. You need to create a file on the class path. For this example, we’ll create a new inside the directory with the following: jets3t.properties jets3t.properties conf/druid/_common s3service.s3-endpoint=localhost
s3service.s3-endpoint-http-port=9000
s3service.disable-dns-buckets=true
s3service.https-only=false Now, comment out the configurations for local storage under the “Deep Storage” section and add appropriate values for Minio. In my case, I’m running MinIO on the same machine as Druid so the S3 Endpoint URL includes localhost and a port. If you’re running on another machine or in a container, you’ll need to change this line to address MinIO in your environment. After this, the “Deep Storage” section should look like: #
# Deep storage
#
# For local disk (only viable in a cluster if this is a network mount):
# druid.storage.type=local
# druid.storage.storageDirectory=var/druid/segments
# For HDFS:
# druid.storage.type=hdfs
# druid.storage.storageDirectory=/druid/segments
# For S3:
druid.storage.type=s3
druid.storage.bucket=druidbucket
druid.storage.baseKey=druid/segments
druid.s3.accessKey= <insert your MinIO accessKey>
druid.s3.secretKey= <insert your MinIO secretKey>
druid.s3.protocol=http
druid.s3.enabePathStyleAccess=true
druid.s3.endpoint.signingRegion=us-east-1
druid.s3.endpoint.url=http://localhost:9000/ We’re also going to configure Druid to store indexing service logs in Minio. It is a good practice to store everything in one place to make it easier to find - Druid is complicated enough without having to search multiple locations for logs. Update the “Indexing service logs” section with appropriate values in . conf/druid/_common/common.runtime.properties After this, the “Indexing service logs” section should look like: # Indexing service logs
# For local disk (only viable in a cluster if this is a network mount):
# druid.indexer.logs.type=file
# druid.indexer.logs.directory=var/druid/indexing-logs
# For HDFS:
# druid.indexer.logs.type=hdfs
# druid.indexer.logs.directory=/druid/indexing-logs
# For S3:
druid.indexer.logs.type=s3
druid.indexer.logs.s3Bucket=druidbucket
druid.indexer.logs.s3Prefix=druid/indexing-logs OK! That covers all of the configuration changes to make to Druid. Save the file and restart your cluster. As you work with Druid, you will see segments and indices written to . druidbucket Installing Apache Superset Apache Superset is a powerful visualization tool that is frequently used with Druid. Superset is a cloud-native business intelligence application that is fast, lightweight, feature-packed, and, best of all, it is free! Superset’s intuitive interface makes crafting interactive dashboards relatively easy. Superset provides out-of-the-box support for most SQL databases. Superset integrates with authentication backends such as the database, OpenID, LDAP and OAuth to secure visualizations. As cloud-native software, Superset is highly scalable and highly available. It was designed to scale out to large, distributed environments. We’re more interested in getting started with Superset than we are with scaling it in production, so we’re going to do the simplest installation we can using . This is the fastest way to try out Superset on Linux or Mac OSX. Docker Compose If you don’t already have Docker installed, please follow . Once you’ve installed Docker, please . Docker’s instructions install Docker Compose Open a terminal window and clone the Superset repo: $ git clone https://github.com/apache/superset.git This will create a superset folder in your current directory. Navigate to that directory and run the following commands (which may take several minutes to complete): $ docker-compose -f docker-compose-non-dev.yml pull
$ docker-compose -f docker-compose-non-dev.yml up You’ll see a whole lot of initialization text pass by on your terminal. Superset is ready once you see something like this: superset_app            | Loaded your LOCAL configuration at [/app/docker/pythonpath_dev/superset_config.py]
superset_app            | 127.0.0.1 - - [12/May/2022:23:59:35 +0000] "GET /health HTTP/1.1" 200 2 "-" "curl/7.74.0" Open a browser and navigate to to access Superset. Log in with the default username “admin” and password “admin”. http://localhost:8088 After logging in to Superset, you must connect it to Druid. On the top right, click the “+” to add a database. From the dropdown list select Apache Druid. Then name the connection and enter the following under SQLALCHEMY URI, making sure to update the IP address or replace it with localhost as necessary. If we had protected the Druid API by requiring authentication, we would also enter a username and password. druid://10.0.0.10:8888/druid/v2/sql Test the connection to make sure it works, and then click Finish. Woo-hoo! You now have Druid, MinIO and Superset running in your environment. With the triumvirate of Druid, SuperSet and MinIO, we can have a complete and timely view of real-time data. There are numerous organizations that have built web-scale data-drive applications on top of this combination for clickstream analytics (web and mobile analytics), risk/fraud analysis, network telemetry analytics (network performance monitoring), supply chain analytics (manufacturing metrics) and application performance metrics. This cloud-native analytics and visualization stack is flexible and extensible, and is scalable and performant so you can build applications that delight your users. Give them the power to monitor, analyze and visualize real-time data. Let them slice and dice, filter and group by any combination of columns so they can make conclusive action-oriented data-driven decisions at the pace business demands. Putting it All Together In order to demonstrate a bit of the functionality we’ve built, we’ll load a tutorial data file of Wikipedia edits over time into Druid and then visualize the edits in SuperSet. Remember, this is backed by MinIO as deep storage. The Wikipedia edits are covered in more detail in . Quickstart · Apache Druid Ingesting into Druid Druid loads data by submitting an ingestion task spec in JSON to the Overlord server. You can build an ingestion spec using the web GUI or write it yourself and schedule it. Once you build and tune an ingestion spec, you can reuse the JSON so you don't have to repeat the task. In production, teams frequently schedule and automate ingestion. Open a browser and load the Druid web interface at . http://localhost:8888 At the top of the window, click Load Data. From here, select Example Data, then select Wikipedia Edits from the drop-down on the right and click the blue Load example button. The web GUI walks you through building an ingestion spec that will read data from a source and store it in segments on deep storage. Ingestion is based on a time stamp that is used to partition and sort data, while data is stored as columns of dimensions and metrics. For more information about how to optimize for performance on ingestion, please see . Data Modeling and Query Performance in Apache Druid With the tutorial data, we’ll simply click through the ingestion workflow accepting the defaults, or click Edit JSON spec and then click Submit. If you wanted (or needed) to, you could look at the raw file as JSON and parse it. Druid creates its own time column called to organize and segment data. Because time is so important to Druid, it’s very good at figuring out how to handle timestamps regardless of their format. In addition, we can transform data, and optimize for query performance by partitioning data and sizing segments. Finally, you can tune the ingestion job in terms of threads and memory utilization, then schedule it and run it. __time The GUI sends you to view Tasks: This is a very small ingestion job so it finishes almost immediately. From here, you could run queries against the data in Druid and get lightning-fast responses. You can run queries via the web GUI or the API. Druid finds itself behind many a dashboard because of its ability to provide excellent query performance over massive amounts of data at scale. Visualizing in Superset Return to the browser where you’ve logged into Superset, and once again click the “+” on the top right of the home screen. Then click Chart. We could run individual SQL queries at the bottom of the screen. Dashboards are groups of individual queries. Responses to repetitive queries are cached in Superset to improve performance. Select the dataset and then select a visualization. Superset includes a number of time-series visualizations. We’re going to choose Time-series Chart and then click New Chart. druid.wikipedia Initially, you won’t see any results, but don’t let that fool you. First, remove the Time Range by setting it to “no filter”. The Time Grain is set too high at “day”, so change that to “5 minute”. We’re now viewing a chart of the number of Wikipedia edits happening in 5 minute windows in our sample data. You can filter and group results using Superset. To save the chart, click Save at the top of the screen, then give the chart a name and save it. By now you’ve learned the basic steps to install Druid and MinIO, ingest your first data set and generate your first Superset chart. A helpful next step could be to learn more about . Exploring Data in Superset Massively Scalable Analytics In this blog post, we introduced a group of cloud-native applications: Apache Druid for analytics, Apache Superset for visualization and MinIO for S3 deep storage. This simple demo is the start of building a cloud-native analytics and visualization stack, and can be extended with Apache Spark for ML or Jupyter for data science. Thanks for reading through this demo. If you have any questions, please send us an email at hello@min.io, or join the and ask away. MinIO slack channel Also published . here

Integrating Apache Druid and Apache Superset with MinIO

Integración de Apache Druid y Apache Superset con MinIO

Intégration d'Apache Druid et d'Apache Superset avec MinIO

अपाचे ड्र्यूड और अपाचे सुपरसेट को मिनिओ के साथ एकीकृत करना

Apache Druid および Apache Superset と MinIO の統合

Integrando Apache Druid e Apache Superset com MinIO

Tích hợp Apache Druid và Apache Superset với MinIO

将 Apache Druid 和 Apache Superset 与 MinIO 集成

My name is Igor Khomyanin, and I'm an experienced Data Scientist and Team Leader. I have a proven track record of delivering impactful results through data-driven decision-making, effective team leadership, and strategic business development in various industries. I'm truly passionate about the world of data science and its potential to drive innovation and growth. In addition to my professional pursuits, I have a strong interest in continuous learning and keeping up with the latest trends in the field. I find immense satisfaction in mentoring and nurturing the growth of aspiring data analysts. This commitment to personal and professional development reflects my belief in the power of collaboration and knowledge-sharing to achieve outstanding results. Outside of work, I enjoy playing video games but still staying active by doing some boxing, rock climbing, and traveling. These experiences provide me with fresh perspectives on the problems I try to solve. Traveling often gives me the inspiration to find unusual approaches to the challenges I encounter in the realm of data science and team leadership. Interesting! What was your latest Hackernoon Top story about? My latest Hackernoon Top story delved into . In this article, I explained the significance of the Bonferroni correction and how it addresses the issue of multiple comparisons in . I discussed its practical application in maintaining the familywise error rate and ensuring more accurate results when conducting multiple tests simultaneously. The positive reception of this article highlights the growing interest in rigorous statistical techniques within the data science community. the intricacies of the Bonferroni correction in statistical analysis hypothesis testing Do you usually write on similar topics? If not, what do you usually write about? While I haven't written extensively before, I'm excited to embark on a new journey of sharing my insights through writing. . I'm eager to dive into topics that showcase how data-driven decision-making can transform industries, drive innovation, and enhance strategic planning. My focus will be on exploring the diverse Applications of Data Science and how they contribute to extracting substantial business value from data My articles will delve into real-world examples of how businesses across various sectors can leverage data science techniques, such as machine learning, statistical analysis, and predictive modeling, to solve complex problems, uncover hidden trends, and make informed decisions. By providing practical insights and actionable recommendations, I hope to contribute to the broader discourse on data-driven approaches that elevate organizational success. Great! What is your usual writing routine like (if you have one?) Write a draft just to (I really mean all of them, you need free space in the brain to produce content). dump all my thoughts on a sheet of paper Polish it until I do not know what to change and improve. Adjourn the process at least for a night. To give my brain time to unconsciously process everything I wrote. Repeat 2 and 3 until I am happy with the result. Being a writer in tech can be a challenge. It’s not often our main role, but an addition to another one. What is the biggest challenge you have when it comes to writing? The biggest challenge I face when it comes to writing is . ensuring that I strike the right balance between depth and accessibility in my articles Explaining complex mathematical or technical concepts in a way that's understandable and relatable to a diverse audience can be a fine art. While it's essential to maintain the integrity of the subject matter, it's equally important to avoid overwhelming readers with jargon or intricate details that might alienate them. My goal is to distill complex ideas into clear and concise explanations that make the content accessible to readers who are eager to learn and apply data-driven techniques but may not have a deep technical background. To overcome this challenge, I invest time in crafting analogies, real-world examples, and relatable scenarios that help convey intricate concepts in a more digestible manner. By focusing on clarity, practicality, and relevance, I aim to create articles that not only educate but also empower readers to take meaningful steps in their own data science journeys. What is the next thing you hope to achieve in your career? I'm excited about the prospect of and collaborating with stakeholders to align data-driven strategies with overarching business goals. This includes driving the integration of advanced analytics and machine learning techniques into decision-making processes, ultimately resulting in more efficient operations, improved customer experiences, and innovative solutions. leading cross-functional teams Furthermore, I plan to within the data science domain. By fostering growth in others and sharing insights from my own journey, I hope to contribute to the growth of the next generation of data scientists and leaders. deepen my involvement in mentoring and nurturing emerging talents Ultimately, my aim is to leave a lasting impact by spearheading data-driven transformations and fostering a culture of continuous innovation and learning within organizations. Wow, that’s admirable. Now, something more casual: What is your guilty pleasure of choice? Eating a big pepperoni pizza with my wife <3 while watching something on Netflix Do you have a non-tech-related hobby? If yes, what is it? Well, in between all the data-driven challenges and strategic thinking, my second guilty pleasure of choice is definitely . It's my way of unwinding and immersing myself in different worlds. There's something uniquely satisfying about diving into epic adventures, solving intricate puzzles, and mastering virtual landscapes. Whether it's exploring fantastical realms or strategizing in competitive environments, video games provide a perfect balance to the analytical nature of my work. It's a delightful escape that lets me recharge and come back to my professional endeavors with renewed energy and creativity. playing video games Outside of the digital realm, I'm also an avid enthusiast of both . These physically demanding activities provide an entirely different kind of satisfaction. Boxing empowers me to channel my determination and focus, while rock climbing offers both a mental and physical challenge that pushes me to overcome obstacles in a literal sense. Just as in data science, both activities require careful strategy, persistence, and the drive to continuously improve. Engaging in these pursuits not only keeps me physically fit but also complements the mental agility needed in my professional life. boxing and rock climbing What can the Hacker Noon community expect to read from you next? Coming up next, the Hacker Noon community can expect some interesting reads from me. . These are cool techniques in the world of data science that help us make better decisions based on cause and effect. I'm considering writing about CUPED or uplift modeling CUPED, for instance, stands for ontrolled Experiments by tilizing re- xperiment ata. This is a technique developed in Microsoft to improve sensitivity of online experiments. I would like to make an easy-to-understand article that people can use a reference to run their first AB test using this statistical trick. C U P E D Uplift modeling takes us beyond the usual average treatment effects often encountered in AB testing. The core concept is to delve into how treatments influence specific units, taking into account various characteristics of each unit. This approach offers the ability to enhance the effectiveness of marketing campaigns, for instance, by selecting clients who will respond favorably only when exposed to treatment, thereby boosting campaign returns. I'm excited to break down these concepts and show how they can be useful in real-life situations. So, keep an eye out for my upcoming articles! What’s your opinion on HackerNoon as a platform for writers? . The editorial support is truly remarkable, ensuring that the content maintains a high standard and is presented in the best possible way. The platform's dedication to promoting top stories gives writers a valuable opportunity to reach a broader audience and receive the recognition they deserve. Additionally, the efficient moderation process allows for swift publication, fostering a dynamic and responsive environment for sharing ideas and insights. Overall, Hacker Noon's commitment to quality, promotion, and timely engagement makes it a valuable platform for writers to showcase their expertise and connect with a like-minded community. I believe Hacker Noon serves as an excellent platform for writers Thanks for taking time to join our “ ” series. It was a pleasure. Do you have any closing words? Meet the writer Thank you for having me in the "Meet the writer" series; it's been a pleasure sharing my thoughts and experiences. To the readers, keep exploring, learning, and pushing boundaries. Let's continue to collaborate and inspire one another in the ever-evolving world of data science and beyond. Looking forward to more insightful discussions and connections within this amazing community. Stay curious and keep reading!

Meet the Writer: Hacker Noon's Contributor Igor Khomyanin, Data Scientist and AI enthusiast

I would like to provide some insights about optimizing code efficiency and memory usage by sharing facts about maps in Go. Once memory is allocated for a map it cannot be freed func mapDelete() {
	v := "some string"

	a := make(map[int]string)

	printMemStats("Before Map Add")

	for i := 0; i < 1000000; i++ {
		a[i] = v
	}

	runtime.GC()
	printMemStats("After Map Add 1000000")

	for i := 0; i < 1000000; i++ {
		delete(a, i)
	}

	runtime.GC()
	printMemStats("After Map Delete 1000000")

	fmt.Printf("%v", a)
}

func printMemStats(mag string) {
	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	fmt.Printf("%v：memory = %vKB, GC Times = %v\n", mag, m.Alloc/1024, m.NumGC)
} Let`s run the code: Before Map Add：memory = 154KB, GC Times = 0
After Map Add 1000000：memory = 56737KB, GC Times = 7
After Map Delete 1000000：memory = 56740KB, GC Times = 8
map[] Wait what? We delete all elements from the map, and now we can see it is empty but go allocates an extra 3KB. The thing is - the key and value are reset to zero when the delete operation on the map is executed, not freeing any space. Maps are always increasing in Go. They can never decrease If you have an excessive amount of elements in a map and want to free space, you can create a new map, copy the elements into it, and there is no need to set the previous map to nil. The garbage collector will delete the map if it is not used again in the program: func mapDeleteWithCopy() {
	v := "some string"

	a := make(map[int]string)

	printMemStats("Before Map Add")

	for i := 0; i < 1000000; i++ {
		a[i] = v
	}

	runtime.GC()
	printMemStats("After Map Add 100000")

	//Just copy the values to the new map
	b := make(map[int]string)
	for i := 0; i < 1000; i++ {
		b[i] = a[i]
	}

	runtime.GC()
	printMemStats("After New Map Copy 1000")

	fmt.Printf("%s", b[0])
}

func printMemStats(mag string) {
	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	fmt.Printf("%v：memory = %vKB, GC Times = %v\n", mag, m.Alloc/1024, m.NumGC)
} As you can see, now we use much less memory: Before Map Add：memory = 155KB, GC Times = 0
After Map Add 100000：memory = 56737KB, GC Times = 7
After New Map Copy 1000：memory = 218KB, GC Times = 8
some string Use struct{} instead of bool as default value Let`s compare the memory consumption of these two examples: func boolDefault() {
	mp := make(map[int]bool)
	for i := 0; i < 10000000; i++ {
		mp[i] = true
	}

	printMemStats("Bool value map")
} Bool value map：memory = 308583KB, GC Times = 8 func structDefault() {
	mp := make(map[int]struct{})
	for i := 0; i < 10000000; i++ {
		mp[i] = struct{}{}
	}

	printMemStats("Struct{} value map")
} Struct{} value map：memory = 280515KB, GC Times = 8 As you can see, we saved 28 megabytes simply by replacing bool with struct{}. And in most cases, this will be enough for our code. Maps are references When you pass a map to a function or assign it to another variable, you are actually passing a reference to the underlying map data. This means any changes made to the map inside the function will affect the original map: func passByReference() {
	mp := make(map[int]string)

	mp[0] = "the dress is blue with black"

	changeElems(mp)

	fmt.Printf("%v", mp)
}

func changeElems(mp map[int]string) {
	mp[0] = "the dress is white with gold"
} Running this code, we get: map[0:the dress is white with gold] As we can see the value of the original map is changed. But what if we need to pass the map as a copy, so the code will not change the original map? We need to copy it. But a simple copy will have no effect: func copyMap() {
	mp := make(map[int]string)

	mp[0] = "the dress is blue with black"

	mp2 := mp

	changeElems(mp2)

	fmt.Printf("%v", mp)
} The original map is changed: map[0:the dress is white with gold] What we need is deep copy: func deepCopyMap() {
	mp := make(map[int]string)

	mp[0] = "the dress is blue with black"

	deepCopiedMap := make(map[int]string, len(mp))
	// Copy from the original map to the new map
	for key, value := range mp {
		deepCopiedMap[key] = value
	}

	changeElems(deepCopiedMap)

	fmt.Printf("%v", mp)
} Now we can see, that the original map is not changed: map[0:the dress is blue with black] Conclusion I tried to explore the interesting details about maps in Go, hoping that it will help you write code more efficiently. In the future, I plan to release more articles about this topic and other things that I find useful for writing efficient code.

Some Things You Should Know About Maps in Go

Personal Recollections of Joan of Arc — Volume 1 by Mark Twain, is part of the HackerNoon Books Series. You can jump to any chapter in this book . Joan Convinces the King here Joan Convinces the King WELL, anything to make delay. The King’s council advised him against arriving at a decision in our matter too precipitately. He arrive at a decision too precipitately! So they sent a committee of priests—always priests—into Lorraine to inquire into Joan’s character and history—a matter which would consume several weeks, of course. You see how fastidious they were. It was as if people should come to put out the fire when a man’s house was burning down, and they waited till they could send into another country to find out if he had always kept the Sabbath or not, before letting him try. So the days poked along; dreary for us young people in some ways, but not in all, for we had one great anticipation in front of us; we had never seen a king, and now some day we should have that prodigious spectacle to see and to treasure in our memories all our lives; so we were on the lookout, and always eager and watching for the chance. The others were doomed to wait longer than I, as it turned out. One day great news came—the Orleans commissioners, with Yolande and our knights, had at last turned the council’s position and persuaded the King to see Joan. Joan received the immense news gratefully but without losing her head, but with us others it was otherwise; we could not eat or sleep or do any rational thing for the excitement and the glory of it. During two days our pair of noble knights were in distress and trepidation on Joan’s account, for the audience was to be at night, and they were afraid that Joan would be so paralyzed by the glare of light from the long files of torches, the solemn pomps and ceremonies, the great concourse of renowned personages, the brilliant costumes, and the other splendors of the Court, that she, a simple country-maid, and all unused to such things, would be overcome by these terrors and make a piteous failure. No doubt I could have comforted them, but I was not free to speak. Would Joan be disturbed by this cheap spectacle, this tinsel show, with its small King and his butterfly dukelets?—she who had spoken face to face with the princes of heaven, the familiars of God, and seen their retinue of angels stretching back into the remoteness of the sky, myriads upon myriads, like a measureless fan of light, a glory like the glory of the sun streaming from each of those innumerable heads, the massed radiance filling the deeps of space with a blinding splendor? I thought not. Queen Yolande wanted Joan to make the best possible impression upon the King and the Court, so she was strenuous to have her clothed in the richest stuffs, wrought upon the princeliest pattern, and set off with jewels; but in that she had to be disappointed, of course, Joan not being persuadable to it, but begging to be simply and sincerely dressed, as became a servant of God, and one sent upon a mission of a serious sort and grave political import. So then the gracious Queen imagined and contrived that simple and witching costume which I have described to you so many times, and which I cannot think of even now in my dull age without being moved just as rhythmical and exquisite music moves one; for that was music, that dress—that is what it was—music that one saw with the eyes and felt in the heart. Yes, she was a poem, she was a dream, she was a spirit when she was clothed in that. She kept that raiment always, and wore it several times upon occasions of state, and it is preserved to this day in the Treasury of Orleans, with two of her swords, and her banner, and other things now sacred because they had belonged to her. At the appointed time the Count of Vendome, a great lord of the court, came richly clothed, with his train of servants and assistants, to conduct Joan to the King, and the two knights and I went with her, being entitled to this privilege by reason of our official positions near her person. When we entered the great audience-hall, there it all was just as I have already painted it. Here were ranks of guards in shining armor and with polished halberds; two sides of the hall were like flower-gardens for variety of color and the magnificence of the costumes; light streamed upon these masses of color from two hundred and fifty flambeaux. There was a wide free space down the middle of the hall, and at the end of it was a throne royally canopied, and upon it sat a crowned and sceptered figure nobly clothed and blazing with jewels. It is true that Joan had been hindered and put off a good while, but now that she was admitted to an audience at last, she was received with honors granted to only the greatest personages. At the entrance door stood four heralds in a row, in splendid tabards, with long slender silver trumpets at their mouths, with square silken banners depending from them embroidered with the arms of France. As Joan and the Count passed by, these trumpets gave forth in unison one long rich note, and as we moved down the hall under the pictured and gilded vaulting, this was repeated at every fifty feet of our progress—six times in all. It made our good knights proud and happy, and they held themselves erect, and stiffened their stride, and looked fine and soldierly. They were not expecting this beautiful and honorable tribute to our little country-maid. Joan walked two yards behind the Count, we three walked two yards behind Joan. Our solemn march ended when we were as yet some eight or ten steps from the throne. The Count made a deep obeisance, pronounced Joan’s name, then bowed again and moved to his place among a group of officials near the throne. I was devouring the crowned personage with all my eyes, and my heart almost stood still with awe. The eyes of all others were fixed upon Joan in a gaze of wonder which was half worship, and which seemed to say, “How sweet—how lovely—how divine!” All lips were parted and motionless, which was a sure sign that those people, who seldom forget themselves, had forgotten themselves now, and were not conscious of anything but the one object they were gazing upon. They had the look of people who are under the enchantment of a vision. Then they presently began to come to life again, rousing themselves out of the spell and shaking it off as one drives away little by little a clinging drowsiness or intoxication. Now they fixed their attention upon Joan with a strong new interest of another sort; they were full of curiosity to see what she would do—they having a secret and particular reason for this curiosity. So they watched. This is what they saw: She made no obeisance, nor even any slight inclination of her head, but stood looking toward the throne in silence. That was all there was to see at present. I glanced up at De Metz, and was shocked at the paleness of his face. I whispered and said: “What is it, man, what is it?” His answering whisper was so weak I could hardly catch it: “They have taken advantage of the hint in her letter to play a trick upon her! She will err, and they will laugh at her. That is not the King that sits there.” Then I glanced at Joan. She was still gazing steadfastly toward the throne, and I had the curious fancy that even her shoulders and the back of her head expressed bewilderment. Now she turned her head slowly, and her eye wandered along the lines of standing courtiers till it fell upon a young man who was very quietly dressed; then her face lighted joyously, and she ran and threw herself at his feet, and clasped his knees, exclaiming in that soft melodious voice which was her birthright and was now charged with deep and tender feeling: “God of his grace give you long life, O dear and gentle Dauphin!” In his astonishment and exultation De Metz cried out: “By the shadow of God, it is an amazing thing!” Then he mashed all the bones of my hand in his grateful grip, and added, with a proud shake of his mane, “Now, what have these painted infidels to say!” Meantime the young person in the plain clothes was saying to Joan: “Ah, you mistake, my child, I am not the King. There he is,” and he pointed to the throne. The knight’s face clouded, and he muttered in grief and indignation: “Ah, it is a shame to use her so. But for this lie she had gone through safe. I will go and proclaim to all the house what—” “Stay where you are!” whispered I and the Sieur Bertrand in a breath, and made him stop in his place. Joan did not stir from her knees, but still lifted her happy face toward the King, and said: “No, gracious liege, you are he, and none other.” De Metz’s troubles vanished away, and he said: “Verily, she was not guessing, she knew. Now, how could she know? It is a miracle. I am content, and will meddle no more, for I perceive that she is equal to her occasions, having that in her head that cannot profitably be helped by the vacancy that is in mine.” This interruption of his lost me a remark or two of the other talk; however, I caught the King’s next question: “But tell me who you are, and what would you?” “I am called Joan the Maid, and am sent to say that the King of Heaven wills that you be crowned and consecrated in your good city of Rheims, and be thereafter Lieutenant of the Lord of Heaven, who is King of France. And He willeth also that you set me at my appointed work and give me men-at-arms.” After a slight pause she added, her eye lighting at the sound of her words, “For then will I raise the siege of Orleans and break the English power!” The young monarch’s amused face sobered a little when this martial speech fell upon that sick air like a breath blown from embattled camps and fields of war, and this trifling smile presently faded wholly away and disappeared. He was grave now, and thoughtful. After a little he waved his hand lightly, and all the people fell away and left those two by themselves in a vacant space. The knights and I moved to the opposite side of the hall and stood there. We saw Joan rise at a sign, then she and the King talked privately together. All that host had been consumed with curiosity to see what Joan would do. Well, they had seen, and now they were full of astonishment to see that she had really performed that strange miracle according to the promise in her letter; and they were fully as much astonished to find that she was not overcome by the pomps and splendors about her, but was even more tranquil and at her ease in holding speech with a monarch than ever they themselves had been, with all their practice and experience. As for our two knights, they were inflated beyond measure with pride in Joan, but nearly dumb, as to speech, they not being able to think out any way to account for her managing to carry herself through this imposing ordeal without ever a mistake or an awkwardness of any kind to mar the grace and credit of her great performance. The talk between Joan and the King was long and earnest, and held in low voices. We could not hear, but we had our eyes and could note effects; and presently we and all the house noted one effect which was memorable and striking, and has been set down in memoirs and histories and in testimony at the Process of Rehabilitation by some who witnessed it; for all knew it was big with meaning, though none knew what that meaning was at that time, of course. For suddenly we saw the King shake off his indolent attitude and straighten up like a man, and at the same time look immeasurably astonished. It was as if Joan had told him something almost too wonderful for belief, and yet of a most uplifting and welcome nature. It was long before we found out the secret of this conversation, but we know it now, and all the world knows it. That part of the talk was like this—as one may read in all histories. The perplexed King asked Joan for a sign. He wanted to believe in her and her mission, and that her Voices were supernatural and endowed with knowledge hidden from mortals, but how could he do this unless these Voices could prove their claim in some absolutely unassailable way? It was then that Joan said: “I will give you a sign, and you shall no more doubt. There is a secret trouble in your heart which you speak of to none—a doubt which wastes away your courage, and makes you dream of throwing all away and fleeing from your realm. Within this little while you have been praying, in your own breast, that God of his grace would resolve that doubt, even if the doing of it must show you that no kingly right is lodged in you.” It was that that amazed the King, for it was as she had said: his prayer was the secret of his own breast, and none but God could know about it. So he said: “The sign is sufficient. I know now that these Voices are of God. They have said true in this matter; if they have said more, tell it me—I will believe.” “They have resolved that doubt, and I bring their very words, which are these: Thou art lawful heir to the King thy father, and true heir of France. God has spoken it. Now lift up thy head, and doubt no more, but give me men-at-arms and let me get about my work.” Telling him he was of lawful birth was what straightened him up and made a man of him for a moment, removing his doubts upon that head and convincing him of his royal right; and if any could have hanged his hindering and pestiferous council and set him free, he would have answered Joan’s prayer and set her in the field. But no, those creatures were only checked, not checkmated; they could invent some more delays. We had been made proud by the honors which had so distinguished Joan’s entrance into that place—honors restricted to personages of very high rank and worth—but that pride was as nothing compared with the pride we had in the honor done her upon leaving it. For whereas those first honors were shown only to the great, these last, up to this time, had been shown only to the royal. The King himself led Joan by the hand down the great hall to the door, the glittering multitude standing and making reverence as they passed, and the silver trumpets sounding those rich notes of theirs. Then he dismissed her with gracious words, bending low over her hand and kissing it. Always—from all companies, high or low—she went forth richer in honor and esteem than when she came. And the King did another handsome thing by Joan, for he sent us back to Courdray Castle torch-lighted and in state, under escort of his own troop—his guard of honor—the only soldiers he had; and finely equipped and bedizened they were, too, though they hadn’t seen the color of their wages since they were children, as a body might say. The wonders which Joan had been performing before the King had been carried all around by this time, so the road was so packed with people who wanted to get a sight of her that we could hardly dig through; and as for talking together, we couldn’t, all attempts at talk being drowned in the storm of shoutings and huzzas that broke out all along as we passed, and kept abreast of us like a wave the whole way. About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Mark Twain (2004). Personal Recollections of Joan of Arc — Volume 1. Urbana, Illinois: Project Gutenberg. Retrieved October 2022 https://www.gutenberg.org/cache/epub/2874/pg2874-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at , located at . www.gutenberg.org https://www.gutenberg.org/policy/license.html

Joan Convinces the King

Deduplication serves a variety of purposes and can be applied in numerous scenarios to address specific challenges effectively. Consider the situation where you possess a vast collection of personal photos stored on your computer. To ensure a backup of these photos, you wish to replicate them onto an external USB HDD. However, due to the immense quantity of photos, they exceed the disk's capacity. While purchasing additional storage is not part of your plan, having some form of backup is crucial. This is where deduplication technology comes into play, offering a viable solution. Alternatively, imagine you have a corporate server housing critical data backups, with another copy stored in a separate office. However, you desire an additional offline backup that can be periodically connected through a NAS device (tape backups may not be feasible for you). The periodic connection is deliberate as it provides protection against potential online threats such as ransomware, enabling you to maintain a "cold" backup within close reach. Nevertheless, the NAS device's capacity is limited to a few terabytes, while the backup itself occupies significantly more space. Furthermore, there are scenarios involving ephemeral VMs deployed by the thousands per day, which individually do not possess important data, but collectively consume considerable disk space due to shared libraries. So, what exactly is deduplication? Deduplication, in the context of data management, refers to the process of identifying and eliminating duplicate data within a file system. By doing so, deduplication reduces overall data size and mitigates storage costs. It's worth noting that deduplication can also be implemented within databases, although that aspect falls beyond the scope of this article. Types and Variants of Data Deduplication: Block-level Deduplication: This method involves identifying redundant data blocks within a file system and retaining only one instance of each unique block. Rather than duplicating these blocks, references to the singular copy are established, resulting in significant savings in disk space. File-level Deduplication: With file-level deduplication, redundancy is assessed at the file level. The system identifies duplicate files and stores only a single copy of each unique file. Instead of creating duplicate files, the system employs references that point to the shared copy. Content-aware Deduplication at the Block Level: Content-aware deduplication analyzes the actual content of data blocks to identify duplicates. It can detect and remove identical blocks, even when they are physically scattered across different locations within the file system or exhibit minor discrepancies. Implementations of data deduplication often combine multiple approaches to achieve optimal results. Additionally, deduplication can be implemented either "inline" or through "post-processing." Inline Deduplication: This technique executes deduplication in real-time as data is being written to the disk. During the process of adding new data to the file system, the system identifies duplicates and eliminates redundant data blocks before storing the data on the disk. This eliminates the need to store duplicate data and instantaneously saves storage space. An excellent example of inline deduplication is when you transfer 10TB of data to a disk that incorporates inline deduplication, resulting in a storage volume of 5TB. Assuming a deduplication ratio of x2 based on your data type. Post-processing Deduplication: In contrast, post-processing deduplication occurs after the data has already been written to the disk. This background process scans the file system, identifies duplicate data blocks, and removes redundant copies. While post-processing deduplication may require more resources, as it entails scanning the entire file system to identify repetitions, it can be highly effective, especially for file systems where data frequently undergoes changes. For instance, suppose you possess a 10TB disk nearing its capacity, and you wish to reclaim storage space without deleting any data. In that case, post-processing deduplication serves as a suitable solution. While other deduplication implementation variations such as Target Deduplication and Source Deduplication exist, these are primarily utilized within specialized software or Storage Area Networks (SANs), making their assessment and verification challenging within a home lab environment. Our lab environment: Virtualbox 7.0.4 Host PC:
Intel Core i7-10510U CPU 1.80GHz / 2.30 GHz
4 CPU
32GB RAM
NVME Samsung SSD 970 EVO Plus 1TB Windows Server 2022 2 vCPU 4GB RAM Ubuntu Server 23.04 ZFS 2 vCPU 4GB RAM Ubuntu Server 23.04 BTRFS 2 vCPU 4GB RAM Zabbix Appliance 6.4.3 2 vCPU 4GB RAM Official documentation can be found here: https://www.zabbix.com/download_appliance Dataset: 103 GB (111,144,359,353 bytes) Within a tar.gz archive, there are backups of website data. When it comes to website backups, deduplication can play a significant role in optimizing storage utilization and ensuring efficient backup processes. Website backups often contain redundant data, especially if multiple backups have been taken over time. Deduplication technology can identify and eliminate duplicate files or blocks within the backup archive, resulting in reduced storage requirements and faster backup operations. By employing deduplication techniques, you can store only unique files or blocks and replace duplicates with references to a single copy. This approach minimizes the overall storage space needed for storing website backups while ensuring that all essential data remains intact. With block-level deduplication, redundant data blocks within the backup archives can be identified, allowing for storing only one instance of each unique block. Consequently, the storage space required for multiple backups can be significantly reduced by eliminating duplicate blocks. Furthermore, file-level deduplication can identify identical files across different backup archives and store only one copy of each unique file. Instead of duplicating files across backups, references or pointers can be used to access the shared copy, saving considerable storage space. In cases where backups contain similar or slightly modified versions of files, content-aware deduplication at the block level becomes valuable. This method analyzes the actual content of data blocks and identifies duplicate blocks, even if they are located in different positions within the backup archives or possess minor differences. Through content-aware deduplication, storage efficiency can be further enhanced by removing redundant blocks and replacing them with references to the unique blocks. Implementing deduplication within website backup archives can significantly reduce the required storage capacity, streamline backup processes, and optimize overall data management. By eliminating duplicate data and retaining only unique files or blocks, deduplication ensures efficient use of storage resources while maintaining the integrity of the backup data. To minimize performance degradation on the host machine, I performed the tests sequentially. This ensured that only the Zabbix VM and the VM undergoing deduplication testing were active at the same time, while the other two VMs were powered off. This approach helped mitigate any potential impact on overall system performance. Linux ZFS Initiating the dataset copy process Now that the dataset has been successfully copied, let's examine the status of the zpool statistics: ~$ zpool list
NAME        SIZE  ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
dedup-pool  119G   104G  15.4G        -     -   2%  87% 1.00x  ONLINE  - The dataset was copied in approximately 18 minutes, but the deduplication process took longer, taking around 30 minutes: Now, let's examine the CPU as well: There is not much disparity in terms of memory utilization. The deduplication process has yielded a disk space utilization of around 33 GB, achieving an impressive deduplication ratio of 3.39! $ zpool list
NAME       SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG  CAP DEDUP HEALTH  ALTROOT
dedup-pool 119G 30.6G 88.4G    -      -      1%   25% 3.39x  ONLINE   - Let's examine the disk space utilization by Zabbix: However, looking at the percentages provides a more accurate representation: Overall, deduplication on the zpool yielded the following statistics:
Initial usage: 104GB
Post-deduplication usage: 30.6GB
Deduplication ratio: 3.39 Infographic: Conclusions. In conclusion, ZFS proves to be an effective solution for optimizing disk space utilization. However, it is crucial to consider the balance between memory and disk resources carefully. Additionally, the type of data being stored and deduplicated plays a significant role in achieving optimal results. By making informed decisions regarding resource allocation and understanding the nature of the data, users can maximize the benefits of ZFS deduplication. Linux BTRFS Initiating the file copying process: $ sudo btrfs fi usage /mnt/btrfs_disk 

Overall:
    Device size:                 120.00GiB
    Device allocated:            107.02GiB
    Device unallocated:           12.98GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        103.73GiB
    Free (estimated):             14.47GiB      (min: 7.98GiB)
    Free (statfs, df):            14.47GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              112.08MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:105.00GiB, Used:103.51GiB (98.58%)
   /dev/sdb      105.00GiB

Metadata,DUP: Size:1.00GiB, Used:113.28MiB (11.06%)
   /dev/sdb        2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
   /dev/sdb       16.00MiB

Unallocated:
   /dev/sdb       12.98GiB As we can see, approximately 103.73GiB is occupied Results: $ sudo btrfs fi usage /mnt/btrfs_disk
Overall:
    Device size:                 120.00GiB
    Device allocated:            108.02GiB
    Device unallocated:           11.98GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                         82.30GiB
    Free (estimated):             35.88GiB      (min: 29.89GiB)
    Free (statfs, df):            35.87GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               97.64MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:106.01GiB, Used:82.11GiB (77.46%)
   /dev/sdb      106.01GiB

Metadata,DUP: Size:1.00GiB, Used:100.41MiB (9.81%)
   /dev/sdb        2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
   /dev/sdb       16.00MiB

Unallocated:
   /dev/sdb       11.98GiB The entire process lasted for more than 7 hours. However, the deduplication results were not as impressive as anticipated. The initially occupied space was 103.73GiB, and after the deduplication process, it reduced to only 82.30GiB, resulting in a modest reduction of approximately 20.66%. Infographic: Conclusions: On the other hand, it is evident that BTRFS is less demanding in terms of memory requirements. However, considering the compression results achieved, the importance of memory utilization becomes less significant for me personally. Windows NTFS Initiating the file transfer process to our system: PS C:\Windows\system32> Get-DedupStatus

FreeSpace    SavedSpace   OptimizedFiles     InPolicyFiles      Volume
---------    ----------   --------------     -------------      ------
16.38 GB     0 B          0                  0                  E:


PS C:\Windows\system32> Get-DedupVolume -Volume E:

Enabled            UsageType          SavedSpace           SavingsRate          Volume
-------            ---------          ----------           -----------          ------
True               Default            0 B                  0 %                  E:


PS C:\Windows\system32> Get-DedupProperties -DriveLetter E


InPolicyFilesCount        : 0
InPolicyFilesSize         : 0
OptimizedFilesCount       : 0
OptimizedFilesSavingsRate : 0
OptimizedFilesSize        : 0
SavingsRate               : 0
SavingsSize               : 0
UnoptimizedSize           : 111247380480
PSComputerName            : Here are the results: PS C:\Windows\system32> Get-DedupStatus

FreeSpace    SavedSpace   OptimizedFiles     InPolicyFiles      Volume
---------    ----------   --------------     -------------      ------
112.85 GB    98.11 GB     377                377                E:


PS C:\Windows\system32> Get-DedupVolume -Volume E:

Enabled            UsageType          SavedSpace           SavingsRate          Volume
-------            ---------          ----------           -----------          ------
True               Default            98.11 GB             93 %                 E:


PS C:\Windows\system32> Get-DedupProperties -DriveLetter E


InPolicyFilesCount        : 377
InPolicyFilesSize         : 111144308918
OptimizedFilesCount       : 377
OptimizedFilesSavingsRate : 94
OptimizedFilesSize        : 111144308918
SavingsRate               : 93
SavingsSize               : 105341122774
UnoptimizedSize           : 112999258326
PSComputerName            : Remarkably, the deduplication results proved to be quite impressive. The data size was reduced by an astonishing 93%! As a result, the occupied disk space now stands at a mere 7.13GB, compared to the initial 103GB. This substantial reduction in data volume not only optimizes storage efficiency but also enables significant savings in precious disk space. Infographic: Results: Deduplication proved to be remarkably efficient for my Windows dataset, offering an exceptional data compression ratio - a true dream! However, it's regrettable that Microsoft has imposed proprietary restrictions and limitations that prevent the utilization of deduplication in the regular version of Windows. Unfortunately, this feature is only available in Windows Server. Deduplication of data is the process of identifying and removing duplicate data within a file system. This helps reduce the amount of disk space used and lowers storage costs. The article discussed various types of deduplication, such as block-level and file-level deduplication, as well as content-aware deduplication at the block level. Deduplication can be implemented either inline or through post-processing. Inline deduplication checks for duplicates and removes them before writing the data to disk, immediately saving space. Post-processing deduplication scans the file system and removes duplicates after the data has been written to disk. Linux ZFS and Windows (on Windows Server) offer effective implementations of data deduplication. For example, Windows achieved an impressive 93% data compression, resulting in significant disk space savings. However, Windows has limitations, and deduplication is only available on Windows Server. Linux BTRFS with BEES showed less impressive deduplication results compared to ZFS and Windows. Although it is less memory-intensive, the space compression results are not as remarkable. The choice between memory and disk resources, as well as the type of data, can impact deduplication effectiveness. Overall, data deduplication can be beneficial in various scenarios, such as creating backups, reducing disk space usage, and optimizing storage resource utilization. However, when selecting a deduplication implementation, it is important to consider the specific characteristics of each system and data type to achieve the best results.

Efficient Data Deduplication: Optimizing Storage Space with NTFS, ZFS, & BTRFS

As developers, we spend a lot of our time in the terminal. Most of the developers I talked to use several tools in their console... yes, the console, that itchy thing that will only display letters and numbers. This tool is more powerful than you think. And there are very many nice handy tools to use. This post outlines my list of tools that every developer must have or has heard about ;). So let's get started! Utils - Auto-correct miss-typed commands thefuck is one of those utilities you won't be able to live without once you've tried it. Whenever you mis-type a command and get an error, just run and it'll auto-correct it. Use up/down to choose a correction, or just run to just execute the most likely immediately. thefuck fuck fuck --yeah - Community-maintained docs ) tldr (better man is a huge collection of community-maintained man pages. Unlike traditional man pages, they're summarized, contain useful usage examples and nicely colourized for easy reading tldr - Count lines of code ) scc (better cloc gives you a breakdown of number of lines of code written in each language for a specific directory. It also shows some fun stats, like estimated cost to develop and complexity info. It's incredibly fast, very accurate and has support for a wide range of languages scc - Listing Files ) exa (better ls is a modern Rust-based replacement for the command, for listing files. It can display file-type icons, colors, file/folder info and has several output formats - tree, grid or list exa ls - Disk Usage ) duf (better df is great for showing info about mounted disks and checking free space. It produces a clear and colorful output, and includes options for sorting and customizing results. duf - Download Utility ) aria2 (better wget is a lightweight, multi-protocol, resuming download utility for HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink, with support for controlling via an RPC interface. It's incredibly , and has tons of . There's also - a nice web interface companion. aria2 feature rich options ziahamza/webui-aria2 - Reading Files ) bat (better cat is a clone of with syntax highlighting and git integration. Written in Rust, it's very performant, and has several options for customizing output and theming. There's support for automatic piping and file concatenation bat cat - File Comparisons ) diff-so-fancy (better diff gives you better looking diffs for comparing strings, files, directories and git changes. The change highlighting makes spotting changes much easier, and you can customize the output layout and colors diff-so-fancy - Watch for changes entr lets you run an arbitrary command whenever file changes. You can pass a file, directory, symlink or regex to specify which files it should watch. It's really useful for automatically rebuilding projects, reacting to logs, automated testing, etc. Unlike similar projects, it uses kqueue(2) or inotify(7) to avoid polling, and improve performance entr - Reading + writing metadata exiftool ExifTool is handy utility for reading, writing, stripping and creating meta information for a wide variety of file types. Never accidentally leak your location when sharing a photo again! - Duplicate file finder fdupes is used for identifying and/or deleting duplicate files within specified directories. It's useful for freeing up disk space when you've got two or more identical files jdupes - Fuzzy file finder ) fzf (better find is an extremely powerful, and easy to use fuzzy file finder and filtering tool. It lets you search for a string or pattern across files. fzf also has available for most shells and IDEs, for showing instant results while searching. This by Alexey Samoshkin highlights some of it's use cases. fzf plugins post - Command benchmarking hyperfine makes it easy to accurately benchmark and compare arbitrary commands or scripts. It takes care of warm-up runs, clearing the cache for accurate results and preventing interference from other programs. It can also export results as raw data and generate charts. hyperfine - Process viewer procs (better ps) is an easy to navigate process viewer, it has colored highlighting, makes sorting and searching for processes easy, has tree view and updates in real-time procs - Search within files ) ripgrep (better grep is a line-oriented search tool that recursively searches the current directory for a regex pattern. It can ignore the contents of and skip binary files. It's able to search within compressed archives, or only search specific extension, and understands files using various encoding methods ripgrep .gitignore - Fast, incremental file transfer rsync lets you copy large files locally or to or from remote hosts or external drives. It can be used to keep files across multiple locations synced, and is perfect for creating, updating and restoring backups rsync - Find and replace ) sd (better sed is an easy, fast and intuitive find and replace tool, based on string literals. It can be executed on a file, an entire directory, or any piped text sd - Directory hierarchy ) tre (better tree outputs a tree stye list of files for your current or a specified directory, with colors. When running with the option, it numbers each item, and creates a temporary alias that you can use to quickly jump to that location tre -e CLI Monitoring and Performance Apps - Bandwidth utilization monitor bandwhich Show bandwidth usage, connection information, outgoing hosts and DNS queries in real-time - Container metrics and monitoring ctop Like , but for monitoring resource usage for running (Docker and runC) containers. It shows real-time CPU, memory and network bandwidth as well as the name, status and ID of each container. There's also a built-in log viewer, and options to manage (stop, start, exec, etc) containers top - Resource monitoring ) bpytop (better htop is a fast, interactive, visual resource monitor. It shows top running processes, recent CPU, mem, disk and network history. From the interface you can navigate, sort and search - there's also support for custom color themes bpytop - Resource monitor + web and API glances is another resource monitor, but with a different feature set. It includes a fully responsive web view, a REST API and historical monitoring. It's easily extendable, and can be integrated with other services glances - Interactive ping tool ) gping (better ping can run ping tests on multiple hosts, while showing results in real-time graph. It can also be used to monitor execution time, when used with the flag gping --cmd - Disk usage analyzer and monitor ) dua-cli (better du let's you interactively view used and available disk space for each mounted drive, and makes freeing up storage easy dua-cli - Command line speed test utility speedtest-cli just runs an internet speed test, via speedtest.net - but straight from the terminal :) speedtest-cli - DNS lookup client ) dog (better dig is an easy-to-use DNS lookup client, with support for DoT and DoH, nicely coloured outputs and the option to emit JSON dog CLI Productivity Apps Surf the web, play music, check emails, manage calendars, read the news and more, all without leaving the terminal! - CLI web browser browsh is a fully interactive, real-time, and modern text-based browser rendered to TTYs and browsers. It supports both mouse and keyboard navigation, and is surprisingly feature rich for a purely terminal based application. It also mitigates battery drain issues that plague modern browsers, and with support for MoSH, you can experience faster load times due to reduced bandwidth browsh - Bookmark manager buku is a terminal-based bookmark manager, with tons of configuration, storage and usage options. There's also an optional and , for accessing your bookmarks outside of the terminal buku web UI browser plugin - Music browser / player cmus is terminal music player, controlled with keyboard shortcuts. It has support for a wide range of audio formats and codecs, and allows organising tracks into playlists and applying playback settings cmus - Track crypto prices cointop show current crypto prices, and track the price history of your portfolio. Supports price alerts, historical charts, currency conversion, fuzzy searching, and much more. You can try the demo via the web at , or by running cointop cointop.sh ssh cointop.sh - Search the web from the terminal ddgr is like , but for DuckDuckGo. It's fast, clean and easy, with support for instant answers, search completion, search bangs, and advanced search. It respects your privacy by default, and also has HTTPS proxy support, and works with Tor ddgr googler - Code editor ) micro (better nano is an easy to use, fast and extendable code editor with mouse support. Since it's packaged into a single binary, installation is as simple as micro curl https://getmic.ro | bash - Calendar client khal is a terminal calendar app, which shows upcoming events, month and agenda views. You can sync it with any CalDAV calendar, and add, edit and remove events directly khal - Email client mutt is a classic, a terminal based mail client for sending, reading and managing emails. It supports all mainstream email protocols and mailbox formats, allows for attachments, BCC/CC, threads, mailing lists and delivery status notifications mut - RSS / ATOM news reader newsboat is an RSS feed reader and aggregator, for reading the news, blogs and following updates directly from the terminal newsboat - Todo + task management taskwarrior is a CLI task management/ todo app. It's both simple and unobtrusive, but also incredibly powerful and scalable, with advanced organisation and query features built in. There's also a lot (700+!) of extra for extending it's functionality and integrating with third-party services task plugins - Terminal UI for Reddit tuir is a great one if you want to look like you're working, while actually browsing Reddit! It's got intuitive keybindings, custom themes, and can render images and multi-media content too. There's also for hacker news tuir haxor CLI Dev Suits - HTTP / API testing testing client httpie is a HTTP client, for testing, debugging and using APIs. It supports everything you'd expect - HTTPS, proxies, authentication, custom headers, persistent sessions, JSON parsing. Usage is simple with an expressive syntax and colourized output. Like other HTTP clients (Postman, Hopscotch, Insomnia, etc) HTTPie also includes a web UI httpie - Full Docker management app lazydocker is a Docker management app, that lets you view all containers and images, manage their state, read logs, check resource usage, restart/ rebuild, analyse layers, prune unused containers, images and volumes, and so much more. It saves you from needing remember, type and chain multiple Docker commands. lazydocker - Full Git management app lazygit is a visual git client, on the command line. Easily add, commit and puch files, resolve conflicts, compare diffs, manage logs, and do complex operations like squashes and rewinds. There's keybindings for everything, colors, and it's easily configurable and extenable lazygit CLI External Sercvices - Reverse proxy for sharing localhost ngrok safely* exposes your localhost to the internet behind a unique URL. This lets you share what you're working on with you're remote colleagues, in real-time. Usage is , but it's also got a lot of advanced features for things like authentication, webhooks, firewalls, traffic inspection, custom/ wildcard domains and much more ngrok very simple - Fast file sharing transfer.sh makes uploading and sharing files really easy, directly from the command line. It's free, supports encryption, gives you a unique URL, and can also be self-hosted.I've written a Bash helper function to make usage a bit easier, you can or try it out by running transfer find it here bash <(curl -L -s https://alicia.url.lol/transfer) - Deploy a site in seconds surge is a free static hosting provider, that you can deploy to directly from the terminal in a single command, just run from within your directory! It supports custom domains, auto SSL certs, pushState support, cross-origin resource support - and it's free! surge surge dist - Check the weather wttr.in is a service that displays the weather in a format that's digestible in the command line. Just run or to try it out. There's URL parameters to customise what data is returned, as well as the format wttr.in curl wttr.in curl wttr.in/London CLI Fun - Have an ASCII cow say your message cowsay is a configurable talking cow. It's based off the by Tony Monroe cowsay original - Output text as big ASCII art text figlet outputs text as ASCII art figlet - Make console output raibow colored lolcat makes any text passed to it rainbow coloured lolcat - Show system data and ditstro info neofetch prints distro and system info (so you can flex that you use Arch btw on r/unixporn) neofetch As an example, I'm using , , and to create a custom time-based MOTD shown to the user when they first log in. It greets them by their name, shows server info and time, date, weather and IP. . cowsay figlet lolcat neofetch Here's the source code Conclusion I think there is at least one interesting tool for you and your productivity enlisted in this list. Hope you enjoy! Also published . here

Stories, News & Mentions about FREESPACE

Freespace’s Stories on HackerNoon