Fiskil

fiskil

Congratulations to all of the participants in our ! You did a wonderful job showcasing your incredible startups and powering through another year. 35,000+ startups and over 600,000+ votes later - it’s time to celebrate the startup scene in ! 2023 Startups of the Year Oceania The startup scene in Oceania is driven by innovation, a bunch of passionate folks, and a supportive environment that's helping cool ventures take off in different industries. With such a vast range of innovative products and ideas, we’re sure it was difficult for you to choose a favorite. 🏆 Key Stats Let’s start out with some general stats for all of you technical folks. Total Startups Participated: 553 Cities Represented: 72 Total Votes Gathered: 168,278 Winning Startups: 32 Without further ado… To see who is a winner in each respective city, visit and pick a city. The name at the top of the list is the winner. startups.hackernoon.com The will announce the winners per region in their respective cities. Startups Winners Announcement Series Now, , here are all the 2023 Startups of the Year winners for : drum roll please Oceania Adelaide, Australia takes the crown with 4 votes! 👑 Mortar Auckland, New Zealand And with 31 votes… takes the title! 👑 Gemini AI Solutions Australian Capital Territory, Australia sweeps the board with 2 votes! 👑 AppSec Fasttrack Belmont, Australia for the win, with 4 votes! 👑 VocaliD Brisbane, Australia takes the crown with 34 votes 👑 Bopple But, we have to give recognition to the runner-up, too! With 5 votes... slides into second place 🥈 Fiskil Canberra, Australia comes out on top with 21 votes! 👑 Knowledge Satnav Christchurch, New Zealand notches a victory with 11 votes! 👑 ioSphere Cremorne, Australia It’s a win for , with 4 votes! 👑 Influx Docklands, New Zealand scores big with 8 votes! 👑 LiteCard Dunedin, New Zealand takes home the glory with 4 votes! 👑 Raum Network Gold Coast, Australia takes the crown with 14 votes! 👑 Tommy Majuro, Marshall Islands seals the deal with 4 votes! 👑 MetaGaming Guild Melbourne, Australia With 3350 votes… clinches the title! 👑 Sharepass New South Wales, Australia The crown goes to , with 14 votes! 👑 Regional CFO New Zealand Misc 12 votes and steals the show! 👑 Odd Jobs Newcastle, Australia scoops the victory with 4 votes! 👑 Cordel Pty North Melbourne, Australia Bagging 21 votes, is the winner! 👑 Peptalk360 Perth, Australia Guess who's the winner? , with 2107 votes! 👑 The Catfisher Project Port Moresby, National Capital District, Papua New Guinea scores the first place with 11 votes! 👑 Alto Trackom Queensland, Australia wins with 21 votes! 👑 Flow Queenstown, New Zealand wins with 4 votes! 👑 Loaded hub South Australia, Australia Air Unique takes the crown with 4 votes! 👑 Surry Hills, Australia With 14 votes, takes the crown! 👑 Zemble Suva, Central Division, Fiji With 11 votes, is Suva’s winner! 👑 eJobsFiji Sydney, Australia takes the cake with 138 votes! 👑 Coviu Thornbury, Australia takes the crown with 11 votes! 👑 Stack Team App Tullamarine, Victoria, Australia is the champ with 2 votes! 👑 HelloTeams Wellington, New Zealand Who won here? With 85 votes... nails it! 👑 Nexustorage Western Australia, Australia takes the crown with 4 votes! 👑 IFN Neurologic Willunga, South Australia, Australia wins with 28 votes! 👑 My Digital CMO Congratulations y’all. We will contact you with details on your winnings! See the global and other regional announcements . here About HackerNoon’s Startups Of The Year Startups of The Year is HackerNoon’s Flagship community-driven event celebrating startups that survived and thrived in 2023. 30,000 startups across 4200+ cities and six continents participated in 2023 to be crowned the best startup in their city. Visit our page to learn more. FAQ Companies that blog more get 67% more leads - Jumpstart your company’s blogging journey by purchasing our . Business Blogging Package The 2023 Startups Of The Year is sponsored by: . . The winners will get a free .Tech Domain, a HackerNoon NFT, and a Tech Company News Page. TECH Domains

Startups of the Year 2023: Oceania Winners

2023 সালের স্টার্টআপ: ওশেনিয়া বিজয়ীরা

Startups des Jahres 2023: Ozeanien-Gewinner

Startups del año 2023: ganadoras de Oceanía

Startups de l'année 2023 : les gagnants d'Océanie

वर्ष 2023 के स्टार्टअप: ओशिनिया विजेता

Startups of the Year 2023: オセアニア受賞者

2023년 올해의 스타트업: 오세아니아 우승자

Startups do ano 2023: Vencedores da Oceania

Стартапы 2023 года: победители Океании

2023 Yılının Startup'ları: Okyanusya Kazananları

Công ty khởi nghiệp của năm 2023: Người chiến thắng ở Châu Đại Dương

2023 年度初创企业：大洋洲获奖者

PLUTONIC AND METAMORPHIC ROCKS:—CLEAVAGE AND FOLIATION

ASCENSION

Both the Apache DolphinScheduler Master and Worker components support multi-node deployment with a decentralized design. The Master is mainly responsible for splitting the DAG workflow and distributing tasks to Worker nodes through RPC, as well as handling task status updates from the Worker.


The worker is responsible for executing tasks and reporting the task status back to the Master, which processes this information. But what happens in case of failure? What if the Master fails? Since it’s responsible for managing workflow instances, the Worker can no longer report task statuses, and the Master can’t process them either.


What if the Worker fails? Since it’s the one executing the actual tasks, how does the Master handle this situation? Let’s dive into the fault tolerance mechanism with the help of an illustration: Fault Tolerance Here’s a breakdown of how DolphinScheduler handles failures: If the Master fails: Other Master nodes will handle the failover using a distributed lock. The workflow instance will switch from the failed Master to a new Master. In this case, the system will issue the new Master’s host address to the Worker, enabling it to report the task status to the new Master. If the Worker fails: The system retries the task. However, before retrying, it needs to kill the task that is still running on YARN. Currently, DolphinScheduler does not support this out-of-the-box, because in a non-client-separated mode, the ProcessBuilder.waitFor method waits for the client process to exit. The applicationId is parsed only after this process exits, meaning the system can only obtain the applicationId after the program has finished running. In other words, you can’t get the applicationIduntil the process is completed. Here’s the relevant code section: org.apache.dolphinscheduler.server.master.service.WorkerFailoverService#killYarnTask 
private void killYarnTask(TaskInstance taskInstance, ProcessInstance processInstance) {
    try {
        if (!masterConfig.isKillApplicationWhenTaskFailover()) {
            return;
        }
        if (StringUtils.isEmpty(taskInstance.getHost()) || StringUtils.isEmpty(taskInstance.getLogPath())) {
            return;
        }
        TaskExecutionContext taskExecutionContext = TaskExecutionContextBuilder.get()
                .buildWorkflowInstanceHost(masterConfig.getMasterAddress())
                .buildTaskInstanceRelatedInfo(taskInstance)
                .buildProcessInstanceRelatedInfo(processInstance)
                .buildProcessDefinitionRelatedInfo(processInstance.getProcessDefinition())
                .create();
        log.info("TaskInstance failover begin kill the task related yarn or k8s job");
        ILogService iLogService =
                SingletonJdkDynamicRpcClientProxyFactory.getProxyClient(taskInstance.getHost(), ILogService.class);
        GetAppIdResponse getAppIdResponse =
                iLogService.getAppId(new GetAppIdRequest(taskInstance.getId(), taskInstance.getLogPath()));
        ProcessUtils.killApplication(getAppIdResponse.getAppIds(), taskExecutionContext);
    } catch (Exception ex) {
        log.error("Kill yarn task error", ex);
    }
} What can be done? In version 1.3.3, the LoggerServer and Masterwere separated, allowing the Master node (if it had the YARN client) to kill the applicationId running on YARN. So what now? Two Possible Solutions: Master kills the task using the YARN REST API: curl -X PUT -d '{"state":"KILLED"}' \
    -H "Content-Type: application/json" \
    http://xx.xx.xx.xx:8088/ws/v1/cluster/apps/application_1694766249884_1098/state?user.name=hdfs Note: You need to specify the user. 2. Worker kills the task:In this case, the task should be marked as a failover task. During retry, the task should be scheduled on a designated Worker node. Before retrying, the runningapplicationId needs to be killed. One optimization would be to first check the YARN status before killing. If the status is abnormal, then kill it. If it's RUNNING, you can wait for a set timeout period. Both the Apache DolphinScheduler Master and Worker components support multi-node deployment with a decentralized design. The Master is mainly responsible for splitting the DAG workflow and distributing tasks to Worker nodes through RPC, as well as handling task status updates from the Worker. The worker is responsible for executing tasks and reporting the task status back to the Master, which processes this information. The Master is mainly responsible for splitting the DAG workflow and distributing tasks to Worker nodes through RPC, as well as handling task status updates from the Worker. The Master is mainly responsible for splitting the DAG workflow and distributing tasks to Worker nodes through RPC, as well as handling task status updates from the Worker. The worker is responsible for executing tasks and reporting the task status back to the Master, which processes this information. The worker is responsible for executing tasks and reporting the task status back to the Master, which processes this information. But what happens in case of failure? But what happens in case of failure? What if the Master fails? Since it’s responsible for managing workflow instances, the Worker can no longer report task statuses, and the Master can’t process them either. What if the Worker fails? Since it’s the one executing the actual tasks, how does the Master handle this situation? What if the Master fails? Since it’s responsible for managing workflow instances, the Worker can no longer report task statuses, and the Master can’t process them either. What if the Master fails? Since it’s responsible for managing workflow instances, the Worker can no longer report task statuses, and the Master can’t process them either. What if the Worker fails? Since it’s the one executing the actual tasks, how does the Master handle this situation? What if the Worker fails? Since it’s the one executing the actual tasks, how does the Master handle this situation? Let’s dive into the fault tolerance mechanism with the help of an illustration: Let’s dive into the fault tolerance mechanism with the help of an illustration: Fault Tolerance Here’s a breakdown of how DolphinScheduler handles failures: If the Master fails: Other Master nodes will handle the failover using a distributed lock. The workflow instance will switch from the failed Master to a new Master. In this case, the system will issue the new Master’s host address to the Worker, enabling it to report the task status to the new Master. If the Master fails: Other Master nodes will handle the failover using a distributed lock. The workflow instance will switch from the failed Master to a new Master. In this case, the system will issue the new Master’s host address to the Worker, enabling it to report the task status to the new Master. If the Worker fails: The system retries the task. However, before retrying, it needs to kill the task that is still running on YARN. Currently, DolphinScheduler does not support this out-of-the-box, because in a non-client-separated mode, the ProcessBuilder.waitFor method waits for the client process to exit. The applicationId is parsed only after this process exits, meaning the system can only obtain the applicationId after the program has finished running. If the Worker fails: The system retries the task. However, before retrying, it needs to kill the task that is still running on YARN. Currently, DolphinScheduler does not support this out-of-the-box, because in a non-client-separated mode, the ProcessBuilder.waitFor method waits for the client process to exit. The applicationId is parsed only after this process exits, meaning the system can only obtain the applicationId after the program has finished running. If the Worker fails: The system retries the task. However, before retrying, it needs to kill the task that is still running on YARN. Currently, DolphinScheduler does not support this out-of-the-box, because in a non-client-separated mode, the ProcessBuilder.waitFor method waits for the client process to exit. The applicationId is parsed only after this process exits, meaning the system can only obtain the applicationId after the program has finished running. ProcessBuilder.waitFor applicationId applicationId In other words, you can’t get the applicationId until the process is completed. applicationId Here’s the relevant code section: org.apache.dolphinscheduler.server.master.service.WorkerFailoverService#killYarnTask 
private void killYarnTask(TaskInstance taskInstance, ProcessInstance processInstance) {
    try {
        if (!masterConfig.isKillApplicationWhenTaskFailover()) {
            return;
        }
        if (StringUtils.isEmpty(taskInstance.getHost()) || StringUtils.isEmpty(taskInstance.getLogPath())) {
            return;
        }
        TaskExecutionContext taskExecutionContext = TaskExecutionContextBuilder.get()
                .buildWorkflowInstanceHost(masterConfig.getMasterAddress())
                .buildTaskInstanceRelatedInfo(taskInstance)
                .buildProcessInstanceRelatedInfo(processInstance)
                .buildProcessDefinitionRelatedInfo(processInstance.getProcessDefinition())
                .create();
        log.info("TaskInstance failover begin kill the task related yarn or k8s job");
        ILogService iLogService =
                SingletonJdkDynamicRpcClientProxyFactory.getProxyClient(taskInstance.getHost(), ILogService.class);
        GetAppIdResponse getAppIdResponse =
                iLogService.getAppId(new GetAppIdRequest(taskInstance.getId(), taskInstance.getLogPath()));
        ProcessUtils.killApplication(getAppIdResponse.getAppIds(), taskExecutionContext);
    } catch (Exception ex) {
        log.error("Kill yarn task error", ex);
    }
} org.apache.dolphinscheduler.server.master.service.WorkerFailoverService#killYarnTask 
private void killYarnTask(TaskInstance taskInstance, ProcessInstance processInstance) {
    try {
        if (!masterConfig.isKillApplicationWhenTaskFailover()) {
            return;
        }
        if (StringUtils.isEmpty(taskInstance.getHost()) || StringUtils.isEmpty(taskInstance.getLogPath())) {
            return;
        }
        TaskExecutionContext taskExecutionContext = TaskExecutionContextBuilder.get()
                .buildWorkflowInstanceHost(masterConfig.getMasterAddress())
                .buildTaskInstanceRelatedInfo(taskInstance)
                .buildProcessInstanceRelatedInfo(processInstance)
                .buildProcessDefinitionRelatedInfo(processInstance.getProcessDefinition())
                .create();
        log.info("TaskInstance failover begin kill the task related yarn or k8s job");
        ILogService iLogService =
                SingletonJdkDynamicRpcClientProxyFactory.getProxyClient(taskInstance.getHost(), ILogService.class);
        GetAppIdResponse getAppIdResponse =
                iLogService.getAppId(new GetAppIdRequest(taskInstance.getId(), taskInstance.getLogPath()));
        ProcessUtils.killApplication(getAppIdResponse.getAppIds(), taskExecutionContext);
    } catch (Exception ex) {
        log.error("Kill yarn task error", ex);
    }
} What can be done? In version 1.3.3, the LoggerServer and Master were separated, allowing the Master node (if it had the YARN client) to kill the applicationId running on YARN. So what now? LoggerServer Master applicationId Two Possible Solutions: Master kills the task using the YARN REST API: Master kills the task using the YARN REST API: curl -X PUT -d '{"state":"KILLED"}' \
    -H "Content-Type: application/json" \
    http://xx.xx.xx.xx:8088/ws/v1/cluster/apps/application_1694766249884_1098/state?user.name=hdfs curl -X PUT -d '{"state":"KILLED"}' \
    -H "Content-Type: application/json" \
    http://xx.xx.xx.xx:8088/ws/v1/cluster/apps/application_1694766249884_1098/state?user.name=hdfs Note: You need to specify the user. Note: You need to specify the user. 2. Worker kills the task: In this case, the task should be marked as a failover task. During retry, the task should be scheduled on a designated Worker node. Before retrying, the running applicationId needs to be killed. One optimization would be to first check the YARN status before killing. If the status is abnormal, then kill it. If it's RUNNING, you can wait for a set timeout period. applicationId

Analyzing Apache DolphinScheduler's Fault Tolerance Mechanism

Stories, News & Mentions about FISKIL

Fiskil’s Stories on HackerNoon