Parameter-Efficient Fine-Tuning for LLM-Based Arabic-to-English Machine Translation

Parameter-Efficient Fine-Tuning for LLM-Based Arabic-to-English Machine Translation Moudjar Amina1, Bahloul Belahcene1, and Aliane Hassina2 1Djilali Bounaama University, [email protected] [email protected] 2CERIST, [email protected] Abstract Large Language Models (LLMs) such as GPT-3, BLOOM, BERT... have revolutionized natural language processing (NLP), particularly in translation. However, fine-tuning these models for downstream tasks, such as Arabic-to-English translation, requires extensive computational resources. Traditional full fine-tuning methods that involve updating all parameters of the model pose significant computational and memory challenges, notably for models with billions of parameters. This study investigates the application of LoRA-based PEFT methods on two chosen models for Arabic to English translation, AraT5 and NLLB-200, with a focus on understanding the trade-offs between computational efficiency and translation quality. Keywords: Large Language models, Efficiency, Computational challenges, Parameter-efficient fine-tuning, Arabic-to-English Machine translation. 1 Introduction The advent of LLMs has revolutionized the field of natural language processing, they have large numbers of parameters and complex architectures to capture intricate language patterns, making them highly effective in translation tasks that need nuanced understanding and generation. Yet, fine-tuning LLMs for specific tasks requires enormous computational resources. Parameter-Efficient Fine-Tuning (PEFT) techniques have emerged to mitigate these challenges by selectively adjusting a small subset of parameters. Among PEFT methods, Low-Rank Adaptation (LoRA) and its variants stand out for their effectiveness, by introducing low-rank matrices to specific layers, allowing the model to learn task-specific adaptations efficiently. Quantized methods go further by quantizing the parameters involved in fine-tuning. Our study focuses on LLMs with encoder-decoder architecture: AraT5 and NLLB-200, which are ideal for Arabic-to-English translation. By applying the previous techniques, we aim to capture the trade-offs between computational efficiency and model performance in machine translation. Our key contributions are: 1. We apply LoRA, DoRA, QLoRA, and QDoRA techniques to fine-tune the AraT5 and NLLB-200 models for Arabic-to-English translation. 2. We assess the computational efficiency of these methods, comparing them to traditional full fine tuning. 3. We evaluate translation quality across the different f ine-tuning methods, examining the trade-offs be tween efficiency and performance. This paper is organized as follows: The first section reviews LLM-based machine translation approaches, the second section details parameter-efficient fine-tuning techniques, the third section presents our methodology, chosen datasets and models, and the fourth section compares resource usage and translation quality across methods and models. 2 Related works Machine Translation has undergone significant transformation recently, primarily due to the rapid advancements in LLMs. These advancements have pushed research into LLM-based machine translation, focusing on two main paradigms: In-Context Learning (ICL) and Finetuning. ICL leverages optimal in-context examples [2] [36] [18] , dictionary knowledge [13] [25], adaptive learning 101 [35] [27], and translation memories [34] to enhance translation accuracy. Traditional machine translation models, particularly those using statistical methods, struggle with contextually rich languages like Arabic, which demand effective capture of long-range dependencies and contextual nuances.Recent advancements in context-aware neural machine translation models, particularly those using self-attention mechanisms, have shown better performance by dynamically focusing on different parts of the input sentence and its context. These models benefit from incorporating larger context windows and external contextual information, such as linguistic annotations and discourse relations, significantly improving the translation quality. Concurrently, finetuning has been instrumental in augmenting LLM’s capability to translate unseen languages and domains [42] [26] and in building multilingual models [44] [46]. Additionally, research has delved into post-editing translation outcomes [28] [33] and utilizing LLMs for machine translation evaluation [12] [11]. The finetuning process for Arabic to English involves adapting pre-trained LLMs, such as T5, to the specific requirements of the translation task. This process includes further training on parallel Arabic-English datasets, which helps the model capture the syntactic, semantic, and contextual intricacies of both languages. Techniques like domain adaptation, advanced regularization methods, back-translation, and self-training enhance the performance and robustness of the finetuned models. The quality and size of the parallel corpus are crucial for achieving high translation accuracy, highlighting the importance of high-quality, diverse, and representative sentence pairs in the training data. Recent advancements in multilingual machine translation have shown significant improvements in performance across various language families, including Afro-Asiatic languages and specific language pairs such as Arabic to English. Zhu et al. (2023) [45] evaluated several LLMs on the FLORES-101 dataset [14] using the SentencePiece BLEU (spBLEU) metric (both SentencePiece BLEU and sacreBLEU are libraries used for calculating BLEU scores). For Afro-Asiatic languages (which include Arabic), LLaMA2-7B [39] achieved the highest BLEU score of 57.72, followed by XGLM-7.5B [22] at 54.51 and Falcon-7B [3] at 38.62. Focusing on the Arabic-English pair, GPT-4 performed better than ChatGPT, with scores of approximately 45 and 40 respectively. These results come from remarkably large models, which explains the high BLEU score results. 3 Parameter-efficient Fine-tuning (PEFT) PEFT adapts an LLM to downstream tasks by freezing the entire LLM backbone and updating only a small set of newly introduced parameters. PEFT methods can be classified into four categories: lowrank adaptation (LoRA [17]), adapter-based tuning (inserting trainable modules into LLMs to simplify fine-tuning [16]), prefix tuning (adding trainable vectors to each LLM layer that are adapted to specific tasks [21]), and prompt tuning (adjusting only the input layer by incorporating trainable prompt tokens that can be placed at the beginning or within the input text [20]). PEFT methods necessitate careful consideration of several factors to balance performance and efficiency effectively. A major challenge lies in minimizing trainable parameters while maintaining Substantial performance [30]. Fine-tuning too few parameters can restrict the model's adaptability to the target task, while excessive fine-tuning can degrade the computational advantages of PEFT [9] [24]. The success of PEFT also relies on the quality and quantity of data available, particularly in domains with limited or noisy data where achieving the same accuracy as full fine-tuning can be tough. In such cases, careful selection of data augmentation techniques and transfer learning strategies is important [8] [4]. 3.1 Low Rank Adaptation (LoRA) Low-Rank Adaptation [17], proposed by Hu et al. (2021), is a widely used Parameter-Efficient FineTuning approach aimed to optimize the adaptation of LLMs to specific tasks. During full fine-tuning, the model is initialized to pre-trained weights W0and updated to W0+ ∆WThe basic hypothesis behind LoRA is that during fine-tuning, A low-rank approximation can powerfully capture the necessary adjustments to the model’s weights. This means that the changes in the weight matrix resulting from task-specific adaptation have a low ”intrinsic rank” [1]. As shown in Figure 3the information contained within the ∆Wmatrix can be represented using fewer dimensions than the original matrix and therefore, the full-dimensional update can be approximated by a product of two smaller matrices while keeping the original weights frozen. 102 Figure 1: Comparison between the traditional fine-tuning approach and the LoRA method 3.1.1 Mathematical Formulation Considering a pre-trained weight matrix Wof a neural network. During fine-tuning, instead of updating Wdirectly, LoRA introduces two trainable low-rank matrices A∈Rm×rand B∈Rr×n. The weight update is then formulated as : Wupdated =W+ ∆W=W+AB (1) Here, ris the rank, a hyperparameter that defines the dimensionality of the low-rank approximation. Using this approach, the amount of trainable parameters is remarkably decreased, as only the matrices. A and Bneed to be learned, while the original weight matrix Wremains frozen. This makes the fine-tuning process much more memory and computation efficient. 3.1.2 Reparametrization and Optimization LoRA modifies the forward pass of the neural network by adding the low-rank update ∆W=AB to the original output. Specifically, if the original output is h=W0x, the updated output becomes: Wupdated =W0x+ ∆W x =W0x+ABx (2) In practice, during backpropagation, the frozen pre-trained weights W0remain untouched, and the loss is only used to update the Band Amatrices introduced by LoRA. Ais initialized with a random Gaussian distribution, while Bis initialized to zero, ensuring that the initial value of ∆Wis zero. The scaling factor αis introduced to balance the contribution of ∆Wduring training, which is crucial for controlling the impact of the low-rank updates and making sure that the fine-tuning process remains stable and effective. 3.1.3 Applying LoRA to a Transformer In a transformer architecture of an LLM, it is more common to apply LoRA to the attention layers because they are computationally expensive and have a significant number of parameters, which makes them the prime targets for parameter-efficient fine-tuning. However, it’s not limited to just attention layers; it could also be applied to other layers like feed-forward networks. LoRA allows the fine-tuning process to require fewer parameters and less computational power while still achieving strong performance. However, as with many advancements in AI, researchers have recently introduced other innovative alternatives and derivatives of LoRA, such as DoRA [23], LoRA+ [15], QA-LoRA [41], QLoRA [6], QDoRA [23], and DyLoRA [40], depending on the model architecture itself and the area of focus. 3.2 Weight-Decomposed Low-Rank Adaptation (DoRA) Weight-Decomposed Low-Rank Adaptation [23] was introduced and built on LoRA by introducing a decomposition of the weight matrix into two components to fine tune them: magnitude and direction, as illustrated in Figure 2. This decomposition separates the fine-tuning of these components, addressing issues in LoRA to make subtle adjustments to weight directions while efficiently handling parameter updates. The process behind DoRA is divided into two main steps. First, the weight matrix W0from a pretrained model is decomposed into two components: 103 Figure 2: An overview of DoRA [23]. •Magnitude Vector m: This vector represents the norm or length of each column in the weight matrix, capturing the scale information. •Directional Matrix V: Where each column vector of the weight matrix is normalized by dividing by its magnitude, keeping only the directional information. Once the pretrained weights are decomposed, and given the substantial size of the directional component in terms of parameters, LoRA is applied exclusively to the directional matrix V, and mis trained as it is, which is feasible because it has just one dimension. DoRA improves both the learning capacity and stability of LoRA, without causing any additional inference overhead. It enables fine-tuning a pretrained model in a way that is computationally efficient and potentially more responsive to new data, maintaining the strengths of the original model while adapting it to new tasks or datasets. 3.3 Quantized Low rank adaptation (QLoRA) QLoRA reduces the memory footprint of LLMs by compressing weights from high-precision data types such as 32-bit floating point to lower-precision formats such as 4-bit integers or NormalFloat, while integrating trainable Low-Rank Adapters (LoRA) for fine-tuning. The pretrained weights remain frozen, and only the LoRA parameters are updated, minimizing memory requirements. During computational tasks, weights stored as 4-bit NormalFloat are dequantized to 16-bit BrainFloat (bfloat16) for both the forward and backward passes. However, only the LoRA parameter's gradients are computed. QLoRA achieves high-fidelity 4-bit fine tuning via two proposed techniques 4-bit NormalFloat (NF4) Quantization and Double Quantization. Paged Optimizers were also introduced to prevent memory spikes during gradient checkpointing from causing out-of-memory errors that have traditionally made fine tuning on a single machine difficult for large models. 3.3.1 4-bit NormalFloat Quantization The NormalFloat (NF) data type builds on Quantile Quantization [5] which is an information-theoretically optimal data type that ensures each quantization bin has an equal number of values assigned from the input tensor. Quantile quantization works by estimating the quantile of the input tensor through the empirical cumulative distribution function. This technique helps compress model weights effectively, but the process of quantile estimation is computationally expensive. Fast quantile approximation algorithms, such as SRAM quantiles, help mitigate this cost, but approximation errors arise, especially for outliers, which are often critical. QLoRA addresses these issues by recognizing that pre-trained LLM weights typically follow a zerocentered normal distribution with a standard deviation α. By transforming all weights to fit within a fixed range (e.g., [−1,1]), accurate quantile estimation becomes feasible, eliminating the need for computationally expensive approximation algorithms. The result is the 4-bit NormalFloat (NF4) data type, which is specifically optimized for normally distributed data. It normalizes neural network weights into this fixed range and quantizes them accordingly, enabling precise weight compression with minimal performance loss. 104 3.3.2 Double Quantization A method that reduces the average memory footprint by quantizing the quantization constants [7]. This saves approximately 0.37 bits per parameter, which translates to around 3 GB for a 65B model. It further reduces the memory overhead of quantization constants. By quantizing both the model weights and the quantization constants, QLoRA achieves higher memory efficiency without negatively impacting model performance. 3.3.3 Paged Optimizers Fine-tuning LLMs can also generate memory spikes, primarily when processing long sequences or large mini-batches. QLoRA harnesses Paged Optimizers to handle these spikes efficiently, leveraging NVIDIA’s unified memory system, which automatically transfers memory between the CPU and GPU. When the GPU runs out of memory, data is paged to the CPU and then moved back to the GPU when needed. This seamless paging mechanism guarantees that memory bottlenecks do not interrupt the training process, allowing QLoRA to handle larger models and batch sizes with fewer resources. 3.4 Quantized Weight-Decomposed Low-Rank Adaptation (QDoRA) QDoRA combines the memory efficiency of QLoRA with the DoRA fine-tuning. QDoRA leverages quantization to compress weights into low-precision formats, significantly reducing memory footprint. However, it goes further by incorporating weight decomposition, as seen in DoRA, to achieve more granular optimization during fine-tuning. This allows QDoRA to maintain both high performance and low computational requirements, making it suitable for fine tuning and training large models like Llama 3 on consumer-grade GPUs. 4 Methodology In this work, we explore the performance of small-sized LLMs in Arabic-to-English machine translation, focusing specifically on encoder-decoder-based architectures. Our research is based on two publicly available LLMs on HuggingFace: the Arabic-focused AraT5v2 Base and the multilingual NLLB-200 distilled-600M models. To evaluate these models, we conducted experiments using the United Nations Parallel Corpus [47] with AraT5v2 [10] and the OPUS-100 corpus [43] with NLLB-200 [38]. Our objective was to evaluate the trade-offs between computational resource efficiency, such as GPU power consumption and memory allocation, and translation performance using BLEU, and ROUGE and Perplexity scores, without an explicit aim to improve translation quality. We applied four fine-tuning techniques in addition to Full fine-tuning (which served as a baseline used for comparison): LoRA, DoRA, and quantized variants: QLoRA and QDoRA. 4.1 Experiments on AraT5v2 with United Nations Parallel Corpus We used the United Nations Parallel Corpus with the AraT5v2 model. We adopted the following dataset splitting strategy to ensure a balanced and effective model training, validation, and testing. We started by loading the first 20 000 examples from the corpus. The dataset was split as into: •Training Set: 15,000 examples (75%) were dedicated to training, ensuring that most of the data is used for model learning. •Validation Set: 2,500 examples (12.5%) were reserved for validation. This validation set is used to track the model’s performance on unseen data, serving it to prevent overfitting. •Test Set: 2,500 examples (12.5%) were set aside for testing. The test set is used for the final evaluation. AraT5v2 built on a foundation set by the original AraT5 model [29]. AraT5 was inspired by the T5 (Text-to-Text Transfer Transformer) model [32], which reframes all NLP tasks into a text-to-text format, offering a unified framework for language modeling tasks. Tokenization is an important step in converting raw text into a format that a model like AraT5v2 can process. AraT5v2 relies on tokenized inputs for both the source language (Arabic) and the target language (English). 105 •Task-Specific Prefix: The T5 model needs a clear prompt of the task it is performing. In this case, we prepend the Arabic input text with the prefix: ”translate Arabic to English: ” •Tokenization Using AraT5v2’s Tokenizer: We use Hugging Face’s AutoTokenizer to tokenize the Arabic input sentences and their corresponding English translations. AutoTokenizer is a generic tokenizer class in the Huggingface Transformers library that automatically selects the proper tokenizer for a given model. The original T5 models (including AraT5, which is based on T5) typically use SentencePiece tokenizers [37], which is a text tokenization algorithm widely used in modern NLP models, particularly in models that need to handle complex languages with rich morphology (like Arabic). Unlike traditional tokenizers that split text based on spaces or punctuation, SentencePiece treats the entire text as a continuous stream of characters and learns how to break it into subword units. •Truncation and Padding: To ensure that the input sequences fit within the model’s constraints, we limit the maximum sequence length to 128 tokens. Sequences longer than this are truncated, while shorter ones are padded to a uniform length. This ensures consistency across batches during training. The tokenized dataset consists of pairs of input and output sequences, where each sequence is a list of tokens representing either an Arabic sentence (input) or its English translation (output). These tokenized sequences are then ready for training the AraT5v2 model. We fine tuned AraT5v2 using different approaches: Full fine-tuning, LoRA, DoRA, and QDoRA: 1. Full fine-tuning means updating all the parameters of the model based on the specific wanted task. We defined several key hyperparameters (used for other techniques as well): Once the training setup Table 1: Hyperparameters Used for Model Training Learning rate 2·10−4 Batch size 2 Number of epochs 5 is complete, the model is trained on the training set by backpropagating through the entire model, including all attention and feed-forward layers, updating every weight in the network. Training is followed by evaluation on the validation set after each epoch to track its performance. 2. We used LoRA to fine-tune AraT5v2, focusing on the following layers involved in the attention mechanism: the key (k), query (q), value (v), and output (o) layers. After freezing the core model parameters, LoRA introduces learnable low-rank matrices that adjust the key, query, value, and output layers of the attention mechanism. These matrices are updated during training, while the original parameters remain untouched. The low-rank structure allows for efficient fine-tuning with fewer parameters to update. 3. In another experiment we applied DoRA by freezing the original model parameters and decomposing the weight matrices involved in the key (k), query (q), value (v), and output (o) layers of the attention mechanism. By applying DoRA to the Target Modules, we decomposed the weight matrices associated with them into their magnitude and directional components. The fine-tuning process updates the directional matrices using LoRA, while the magnitude vector is trained directly. After training, all the components are recombined to form the updated weight matrices which is used for inference. 4. Lastly, QDoRA is applied to the AraT5v2 model by combining two techniques: quantization and Weight Decomposed Low-Rank Adaptation (DoRA). We applied these techniques to the model by quantizing the model’s weights to 4-bit precision, then freezing the core weights of the model (from its pre-trained state), the Decomposition of Weights, LoRA is then applied specifically to the directional matrices within the attention layers chosen, and alongside the updates to the directional matrices, the magnitude vector (which represents the scale of the weights) is also fine-tuned. Since the magnitude has significantly fewer parameters, it can be trained directly without requiring the same low-rank approximations used for the directional matrices. Table 3summarizes the previous techniques. 106 4.2 Experiments on NLLB-200 with OPUS-100 Corpus The NLLB-200 distilled-600M model was fine-tuned using the OPUS-100 dataset. We splitted the dataset into model training, validation, and testing. We started by loading the first 11200 examples from the corpus. The dataset splitting was done as following: •Training Set: 8000 examples (71%) were dedicated to training. •Validation Set: 1600 examples (almost 15%) were reserved for validation. •Test Set: 1600 examples (almost 15%) were set aside for testing. The NLLB-200-distilled-600M represents a distilled version of the full NLLB-200 model of 600 million parameters. NLLB-200 is an innovative solution to the complex challenges of multilingual machine translation, especially in low-resource languages. We use a fast tokenizer NllbTokenizerFast to process the text, specifying Arabic as the source language (src lang) and English as the target language (tgt lang). It is based on the BytePairEncoding [31], which preserves common words in their full form, while splitting less frequent words into subword units, achieving a balance between vocabulary size and representational efficiency. The tokenizer also handles truncation, ensuring that sentences are cut off at the model’s maximum input length to avoid issues during training. By tokenizing the entire dataset, we prepare it for the subsequent steps. Once the dataset is prepared, we move on to the fine-tuning stage: 1. Full fine-tuning allows all the layers of the model to be updated during training. The training is configured controlling various aspects (same configuration for other fine tuning methods of the model): Table 2: Hyperparameters Used for Model Training Learning rate 2·10−5 Batch size 1 Number of epochs 3 Evaluation is performed at the end of each epoch. 2. We applied LoRA where instead of updating all the model’s parameters, only the low-rank matrices are updated during fine-tuning, which drastically reduces the number of parameters that need to be trained. In our case, the target layers for LoRA are:Query projection (q proj), Value projection (V proj). Only the low-rank matrices in the q proj and v proj layers are updated, while the rest of the model remains frozen. This reduces the number of trainable parameters while retaining enough capacity to specialize for the translation task. 3. In another experiment, we applied QLoRA which is an advanced version of the original LoRA technique (same as QDoRA). QLoRA focuses on fine-tuning specific layers of the model; this approach extends the benefits of LoRA by further reducing the model’s memory footprint through quantization, while still fine-tuning only a small fraction of the model’s parameters. The core difference in QLoRA is the use of 4-bit quantization that reduces memory requirements even further while maintaining precision for computation. 4. In another experiment, DoRA was applied to fine-tune the NLLB-200 distilled 600M model for Arabic to English translation. By introducing a decomposition of the weight matrix into magnitude and direction, DoRA enhances the fine-tuning process. DoRA allows for independent updates to the scale and directional aspects of the model’s weights. The directional matrix is fine-tuned using LoRA’s low-rank matrices, while the magnitude vector is directly trained. Table 4summarizes these techniques. 4.3 Evaluation To track CPU, GPU and Memory usage, we used Weights & Biases, a platform for machine learning developers to help them build better models faster. It provides lightweight, interoperable tools for 107 Table 3: AraT5v2 Fine-Tuning Methods Fine-Tuning Parameters Low-Rank Scaling Dropout Quantization DoRA Full Method Updated Dimension Factor (lora dropout) Enabled Parameters (r) (lora alpha) Updated Full All - - - No No Yes Fine-Tuning Parameters LoRA Attention 5 32 0.06 No No No Layers (k, q, v, o) DoRA Magnitude + 5 32 0.06 No Yes No Direction QDoRA Quantized + 5 32 0.06 Yes Yes No Magnitude + Direction Table 4: NLLB200-600M Fine-Tuning Methods Fine-Tuning Parameters Low-Rank Scaling Dropout Quantization DoRA Full Method Updated Dimension Factor (lora dropout) Enabled Parameters (r) (lora alpha) Updated Full All - - - No No Yes Fine-Tuning Parameters LoRA Attention 8 32 0.1 No No No Layers (q, v) DoRA Magnitude + 8 32 0.1 No Yes No Direction QLoRA Quantized + 8 32 0.1 Yes No No Attention Layers (q, v) tracking experiments, versioning datasets and models, evaluating model performance, and visualizing results.Additionally to the W&B system tracking, we evaluated the optimization techniques using Perplexity, that measures the model’s uncertainty in generating the next token, the lower the value the better. In addition, we also used the following evaluation metrics: ROUGE-L, which measures the overlap of longest common subsequences between the system output and reference texts, indicating the quality of text summarization or other generation tasks. And SacreBLEU score, which is an extended implementation that builds upon the basic BLEU metric and provides additional features, such as multiple reference support, better handling of tokenization, and more fine-grained control over the evaluation process. 5 Results and discussion 5.1 Models efficiency comparison Using W&B SDK, we tracked system metrics for every technique on both chosen models. 5.1.1 AraT5v2 Base In full fine tuning, all the parameters are trained and updated, while Table 5presents trainable Parameters in the other techniques. Figures 3and 4araT5v2 variations showcase varying trade-offs between Table 5: Comparison of the Trainable Parameters in AraT5v2 Optimization Trainable Parameters Total Parameters Percentage of Trainable Parameters LoRA 1,105,920 368,614,656 0.30% DoRA 1,105,920 368,614,656 0.30% QDoRA 1,216,512 368,725,248 0.33% power consumption, speed, and resource usage. The full fine tuning of araT5v2 is the fastest, completing 108 training in 2 hours but consumes high power (around 120W) due to its computational intensity. In contrast, araT5v2-Base-with-LoRA balances speed and resource efficiency, using moderate power (50-60W) while finishing second fastest , making it suitable for scenarios where moderate optimization is needed without heavy resource use. Although araT5v2-Base-with-DoRA was slower than full fine tuning and LoRA, DoRA prioritizes power efficiency over speed by applying more optimized updates (magnitude and direction), making it ideal when power usage is a concern and training time is flexible. Lastly, araT5v2-Base-with-QDoRA is the most power-efficient (60W) but takes over 10 hours to train, which makes it useful for resource-constrained settings where power efficiency is extremely crucial. GPU memFigure 3: GPU power usage in Watt in araT5v2 Figure 4: GPU power usage in % in araT5v2 ory directly impacts real-time performance, determining factors like batch size, computation speed, and 109 Figure 15: NLLB200-600M-QLoRA system CPU utilization Figure 16: Bar Chart showing comparative araT5v2 Evaluation Using ROUGEL, SacreBLEU, in addition to Perplexity Figure 17: Bar Chart showing comparative NLLB200-600M Evaluation Using ROUGEL, SacreBLEU, in addition to Perplexity According to Figure 17, The full fine-tuning method starts at a higher ROUGEL and SacreBLEU 116 scores and remains ahead of all parameter-efficient methods. DoRA and LoRA show almost identical performance, with DoRA slightly edging out. QLoRA remains the lowest performer. The perplexity shows that the uniformity of perplexity scores across all methods shown here is notable. It implies that, in terms of understanding and predicting the structure of the target language, all methods are equally effective. Full fine-tuning offers the best evaluation scores results, but parameter-efficient methods like DoRA and LoRA are closely following, suggesting they provide a competitive trade-off between performance and resource usage. 5.2.2 Human evaluation results In our inference tests, Arabic sentences were translated using AraT5v2 and compared with Google Translate. For simple sentences, all fine-tuning methods (Table 7) performed similarly, showing that lightweight techniques like LoRA and QDoRA can handle basic translations effectively. However, errors in complex sentences - especially with idiomatic expressions - revealed significant limitations. Google Translate generally outperformed the fine-tuned models in these cases, delivering more fluent translations, while the models often struggled with figurative language, producing more literal or incomplete outputs. Noticeably, some Arabic words remained untranslated across all fine-tuning methods, indicating gaps in vocabulary handling. Full Fine-Tuning provided the best overall performance but occasionally overcomplicated translations, likely due to overfitting. For NLLB200-600M (Table 8), all fine-tuning methods closely matched Google Translate on straightforward sentences but diverged on abstract phrases. Overall, Full Fine-Tuning excelled but was resourceheavy, while LoRA and DoRA offered a balanced trade-off, and QLoRA proved best suited for extreme memory constraints but was slower to converge. Table 9 presents comparison between GPT models BLEU scores and ours, due to hardware limitations, we were unable to use similarly powerful models in our experiments, resulting in comparatively lower performance to the results we presented in the related works. 6 Conclusion This study explored the trade-offs between computational efficiency and Arabic-to-English translation quality in LLMs, focusing on parameter-efficient fine-tuning techniques, particularly variations of LoRA. While full fine-tuning showed optimal translation accuracy, LoRA and DoRA achieved comparable quality with reduced computational costs remarkably. QLoRA offered additional memory efficiency, though at the expense of longer training time. Future work will explore scaling these methods to larger models and enhancing their capacity to capture complex linguistic structures. All code and models are provided in a GitHub repository as open source, editable Jupyter notebooks. 117 Table 7 Translations generated by araT5v2 Arabic original text DoRA Full fine tuning LoRA QDoRA Google Translate  à @ HPQ ¯ ú  æ ®J  £ð Q  « @  á«  Im' . B  èYK Yg .Q ¯ I have decided to change my job to find new opportunities I decided to change my job to seek new opportunities I decided to change my job to find new opportunities. I decided to change my job to find new opportunities. I decided to change my job to look for new opportunities  J  Ë@  ZA J . K@      Y   @ ú  ¯ I .  J ºË@  áÓ   á K . YmÌ'@ èY g I . ª ÊË@ð Y m.Ì'@ The spend of books is stronger from the table of books, in the same line between the study and the game The cost estimate provides for the highest number of newspapers at the extent of The spend of books is expenditure at the line of the line between the and game. The printing of books are expenditure to the time and the game. The sword is more truthful than books in its sharpness, the line between seriousness and play.  J Ë@ ZAJ . K@  Y@ ú  ¯ I . JºË@  áÓ  á K .YmÌ'@ èYg I .ªÊË@ð Ym.Ì'@  J Ë@ is the most accurate of books, in the scale of the time and game. The sword is the most best source of books at the time of serious and play The  J Ë@ is most accurate of books in the same line between serious and game. The  J Ë@ is the best evidence of books in the same line between the argument and play The sword is more truthful than books in its sharpness, the limit between seriousness and play Table 8 Translations generated by NLLB200-600M Arabic original text DoRA Full fine tuning LoRA QLoRA Google Translate  à @ HPQ ¯ ú  æ ®J  £ð Q  « @  á«  Im' . B  èYK Yg .Q ¯ I decided to change jobs to look for new opportunities. I decided to change jobs to look for new opportunities I decided to change jobs to look for new opportunities. I decided to change jobs to look for new opportunities. I decided to change my job to look for new opportunities  J  Ë@  ZA J . K@      Y   @ ú  ¯ I .  J ºË@  áÓ   á K . YmÌ'@ èY g I . ª ÊË@ð Y m.Ì'@ The sword I believe is a book narrative about the boundary between grandfather and play. The sword. I believe a prophecy from a book alone about the boundary between seriousness and play I believe the sword is a book narrative about the boundary between grandfather and play. I believe the sword is a book narrative about the boundary between grandfather and play. The sword is more truthful than books in its sharpness, the line between seriousness and play  J Ë@ ZAJ . K@  Y@ ú  ¯ I . JºË@  áÓ  á K .YmÌ'@ èYg I .ªÊË@ð Ym.Ì'@ The sword believes prophecies from books about the boundary between grandpa and play. The sword believed the prophecies of the books in the boundary between grandfather and play. The sword believes prophecies from books about the boundary between grandfather and play. Sword believes prophecies from books about the boundary between grandfather and play. The sword is more truthful than books in its sharpness, the limit between seriousness and play Table 9 BLEU Scores for GPT Language Models compared to araT5v2 Base and NLLB200-600M Model BLEU score GPT-4 ∼45 ChatGPT ∼40 araT5v2 Full fine tuning 19.951 araT5v2 Full LoRA 12.531 araT5v2 Full DoRA 13.006 araT5v2 Full QDoRA 13.027 NLLB200-600M Full fine tuning 34.245 NLLB200-600M LoRA 32.676 NLLB200-600M DoRA 32.812 NLLB200-600M QLoRA 31.595 118 References [1] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, 2020. [2] Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. Incontext examples selection for machine translation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 8857–8873, Toronto, Canada, July 2023. Association for Computational Linguistics. [3] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, M´erouane Debbah, ´ Etienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023. [4] Golla Anjali, Santosh Sanjeev, Akuraju Mounika, Gangireddy Suhas, G. Pradeep Reddy, and Yarlagadda Kshiraja. Infant cry classification using transfer learning. In TENCON 2022 - 2022 IEEE Region 10 Conference (TENCON), pages 1–7, 2022. [5] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization, 2022. [6] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. [7] Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws, 2023. [8] Raman Dutt, Ondrej Bohdal, Sotirios A. Tsaftaris, and Timothy Hospedales. Fairtune: Optimizing parameter efficient fine tuning for fairness in medical image analysis, 2024. [9] Raman Dutt, Linus Ericsson, Pedro Sanchez, Sotirios A. Tsaftaris, and Timothy Hospedales. Parameter-efficient fine-tuning for medical image analysis: The missed opportunity, 2024. [10] AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. Octopus: A multitask model and toolkit for Arabic natural language generation. In Hassan Sawaf, Samhaa El-Beltagy, Wajdi Zaghouani, Walid Magdy, Ahmed Abdelali, Nadi Tomeh, Ibrahim Abu Farha, Nizar Habash, Salam Khalifa, Amr Keleg, Hatem Haddad, Imed Zitouni, Khalil Mrini, and Rawan Almatham, editors, Proceedings of ArabicNLP 2023, pages 232–243, Singapore (Hybrid), December 2023. Association for Computational Linguistics. [11] Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, Andr´e F. T. Martins, Graham Neubig, Ankush Garg, Jonathan H. Clark, Markus Freitag, and Orhan Firat. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation, 2023. [12] Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire, 2023. [13] Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. Dictionary-based phrase-level prompting of large language models for machine translation, 2023. [14] Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzm´an, and Angela Fan. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538, 2022. [15] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models, 2024. [16] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790–2799. PMLR, 09–15 Jun 2019. 119 [17] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. [18] Vivek Iyer, Pinzhen Chen, and Alexandra Birch. Towards effective disambiguation for machine translation with large language models, 2023. [19] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning, 2019. [20] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021. [21] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021. [22] Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. Few-shot learning with multilingual language models, 2022. [23] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, KwangTing Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation, 2024. [24] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too, 2023. [25] Hongyuan Lu, Haoran Yang, Haoyang Huang, Dongdong Zhang, Wai Lam, and Furu Wei. Chainof-dictionary prompting elicits translation in large language models, 2024. [26] Zhuoyuan Mao and Yen Yu. Tuning llms with contrastive alignment instructions for machine translation in unseen, low-resource languages, 2024. [27] Yasmin Moslem, Rejwanul Haque, John D. Kelleher, and Andy Way. Adaptive machine translation with large language models. In Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escart´ın, Mikel Forcada, Maja Popovic, Carolina Scarton, and Helena Moniz, editors, Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 227–237, Tampere, Finland, June 2023. European Association for Machine Translation. [28] Yasmin Moslem, Gianfranco Romani, Mahdi Molaei, John D. Kelleher, Rejwanul Haque, and Andy Way. Domain terminology integration into machine translation: Leveraging large language models. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation, pages 902–911, Singapore, December 2023. Association for Computational Linguistics. [29] El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. Arat5: Textto-text transformers for arabic language generation, 2022. [30] Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models, 2024. [31] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. [32] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. [33] Vikas Raunak, Amr Sharaf, Yiren Wang, Hany Awadalla, and Arul Menezes. Leveraging GPT-4 for automatic translation post-editing. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12009–12024, Singapore, December 2023. Association for Computational Linguistics. 120 [34] Abudurexiti Reheman, Tao Zhou, Yingfeng Luo, Di Yang, Tong Xiao, and Jingbo Zhu. Prompting neural machine translation with translation memories, 2023. [35] Raphael Reinauer, Patrick Simianer, Kaden Uhlig, Johannes E. M. Mosig, and Joern Wuebker. Neural machine translation models can learn to be few-shot learners, 2023. [36] Gabriele Sarti, Phu Mon Htut, Xing Niu, Benjamin Hsu, Anna Currey, Georgiana Dinu, and Maria Nadejde. RAMP: Retrieval and attribute-marking enhanced prompting for attribute-controlled translation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1476–1490, Toronto, Canada, July 2023. Association for Computational Linguistics. [37] Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152, 2012. [38] NLLB Team, Marta R. Costa-juss`a, James Cross, Onur C¸elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm´an, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling humancentered machine translation, 2022. [39] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. [40] Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation, 2023. [41] Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models, 2023. [42] Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages, 2023. [43] Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilingual neural machine translation and zero-shot translation, 2020. [44] Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models, 2023. [45] Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual machine translation with large language models: Empirical results and analysis, 2024. [46] Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Extrapolating large language models to non-english by aligning languages, 2023. 121 [47] Micha l Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. The United Nations parallel corpus v1.0. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534, Portoroˇz, Slovenia, May 2016. European Language Resources Association (ELRA). 122